Skip to content

Leave no nvx folders behind, and retry a microVM OpenVMM couldn't start - #978

Merged
lavaman131 merged 1 commit into
mainfrom
nvx-cleanup-retry
Nov 21, 2026
Merged

lavaman131 merged 1 commit into
mainfrom
nvx-cleanup-retry

Conversation

@lavaman131

@lavaman131 lavaman131 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #975. It does the two things #975 listed as not done: nvx sandbox run's temporary folders are no longer left behind, and a microVM start that OpenVMM fails under heavy load is tried again.

Leftover folders

nvx sandbox run keeps a private folder in the temporary directory for each run (nvx-sandbox-run-<id>). It removes the folder itself only when the workload ends on its own. The connector ends a launch it stops by force (a cold Planner on Ctrl-C, for example), so each such stop left a folder holding OpenVMM's log. This happens on Linux and on Windows.

nvx sandbox run won't put the folder anywhere else: it rejects --state-dir. Pointing TMPDIR into the state directory doesn't work either, because OpenVMM's control socket path would go past Linux's 108-byte limit. So the connector now:

  • reads the folder's path from the OpenVMM command line nvx prints (--microvm-report <folder>/outcome.json, runFolderOf) and records it in the launch's launch.json;
  • removes the folder when the launch ends, however it ends. On Windows it tries again for a few seconds while OpenVMM still holds the log;
  • removes the folder named in a dead connector's records along with those records (endOrphans);
  • when chopin connect starts, removes the leftovers no launch record names (sweepRunFolders). It looks where nvx creates them: $TMPDIR or /tmp on Linux (not $TMP), and GetTempPath on Windows. It keeps a folder that belongs to another user, is less than a minute old, or is named by a running process. OpenVMM's command line names its folder for the whole run, so this covers a microVM of another chopin on another base dir, or any other NVX user's. The connector reads command lines from /proc on Linux and from Win32_Process on Windows; if it can't read them, it keeps every folder.

The connector's own files are also out of the shared temporary directory:

  • The copy of the person's git configuration (formerly chopin-git-*) is now in <base>/git/<pid>/. It is removed when that process exits, and chopin connect removes any folder whose process is gone (sweepGitFolders).
  • The agent's start check (formerly chopin-check-*, or <base>/check-* for service install) now runs in one empty folder, <base>/nvx/check.

Starting a microVM again

Under heavy load OpenVMM sometimes fails at startup. Its log ends:

fatal error: failed to launch vm worker
Caused by:
    0: failed to launch worker
    1: failed to resolve resource of type virtio:virtio-blk
    2: failed to resolve resource of type disk_handle:block
    3: failed to create new device
    4: driver does not support io-uring

nvx reports only vmm-failure status 1. The cause is io_uring_setup failing for OpenVMM's block driver. Refusing that one syscall with strace -e inject=io_uring_setup:error=ENOMEM produces exactly this log, so the real-KVM tests use it to cause the failure.

startVm now returns one launch that may start nvx up to three times. It waits 0.5 s before the second start and 1.5 s before the third. It retries only when both of these hold:

  • the start failed with that line in OpenVMM's log. The connector opens the log as soon as nvx prints the folder: nvx creates the log before printing the line and deletes it when a start fails, and an open file descriptor keeps it readable;
  • the workload had not started. The guest's launcher prints >> chopin: the workload starts on stdout just before it execs the agent (STARTED_LINE). Until that line arrives, the launch sends nothing of its stdin to the guest and passes nothing of nvx's stderr to the caller. Once it arrives, the connector strips it, as it already strips nvx's own launch line. So an agent never sees two starts, and never gets input written to a start that failed.

A stop during the wait is honored:

  • On Linux the wait runs as sleep inside the next start's own process group. The caller's group signal ends that group, and the launch ends without starting again. stopBackend re-reads the group each time it polls, in case a start fails just as it is stopped.
  • On Windows callers stop a launch through stopMicrovm, which cancels the wait.

A start that fails for good, either after three tries or at once for any other reason, writes one stderr line with the end of OpenVMM's log. That line is what a run or Planner failure shows:

The microVM didn't start after 3 tries: fatal error: failed to launch vm worker / Caused by: / 0: failed to launch worker / … / 4: driver does not support io-uring (nvx: error[exited]: OpenVMM exited during startup: vmm-failure status 1)

chopin connect reports each new start in its output. The CI microVM job installs strace if the runner lacks it.

Invariants and their tests

Invariant Tests
A launch leaves nothing of nvx's in the temporary directory, however it ends sandbox.test.ts: a launch ended by force leaves no folder, and a dead connector's record takes its folder. nvx-vm.test.ts (KVM): a launch SIGKILLed while running leaves neither its recorded folder nor its record
The sweep never takes a live launch's folder sandbox.test.ts: a folder survives a sweep by another connector on the same base dir (it is in a record) and on another base dir (a process names it); a young folder survives; a stale folder is removed
An agent sees one start sandbox.test.ts with a fake nvx: one io-uring failure then a start reads its stdin once; another failure is not retried; a start whose workload had started is not retried; three starts at most; a stop during the wait starts nothing. nvx-vm.test.ts (KVM, strace): the workload runs and reads its input once
Every start failure says the end of OpenVMM's log sandbox.test.ts (KVM log and io-uring log); nvx-vm.test.ts (three real failures)
The connector's own files are out of the temporary directory git-host.test.ts (the copy is in <base>/git/<pid>, and the sweep removes only folders of dead processes); main.test.ts (the start check runs in <base>/nvx/check)

Each new behaviour test failed when its guard was removed: without the stdin gate, without the removal on end, without the process check, and without both started guards.

Verification

bun run fix, bun run ci, bun run types and the connector unit tests ran on the final head 4ad0f949 (on 6f7a699e). The KVM tests and the connector-related browser tests ran on 88aa64bc, the same change on fca5f0b0. The Windows checks and the first browser run were made before the rebase onto fca5f0b0:

  • bun run fix made no changes. bun run ci and bun run types passed.
  • bun test apps/connector: 431 pass, 29 skip, 0 fail.
  • Real microVMs on this machine's KVM, CHOPIN_NVX_ROOT=<nvx-0.1.0-linux-kvm> bun test apps/connector/src/nvx-vm.test.ts: 12 pass, including the three new tests.
  • Windows (WHP, nvx-win), running this branch's startVm and sweep with the real nvx.exe:
    • A launch that ends by itself has its launch line and started line stripped, gets its stdin after the start, and leaves no folder.
    • A forced stop (job killed at once) and a kill() remove the folder. While a VM runs, its folder holds only openvmm.log. The control channel there is a named pipe, so the folder appears only in --microvm-report.
    • A graceful stop leaves nothing.
    • The sweep through Win32_Process (0.76 s) removes a stale folder and keeps one a process names and a young one. It also removed the six real leftovers from earlier runs in %TEMP%.
    • A start that fails (memory too large for the page file) says The microVM didn't start: fatal error: failed to launch vm worker / … / The paging file is too small for this operation to complete. (os error 1455) (nvx: …) and is not retried.
  • Browser tests, under flock with their own port base and PostgreSQL container:
    • Connector-related set (connector, setup-agents, machines, machine-choice, implementation-launcher, planner-spare, experiments, planner-picker, handover, build-restart, connection-banner, saved-decision-investigation): 148 passed and 16 failed in one run.
    • The implementation-launcher failure passed on a rerun.
    • The other 15 are connection-banner and handover tests of a remote Planner, which don't run the connector. In this environment they get no-ai-account, and two of them fail identically on origin/main without this change.
    • Rerun on 88aa64bc without those two files: 144 passed, 0 failed.

`nvx sandbox run` keeps a folder in the temporary directory for each run
and removes it only when its workload ends by itself. The connector ends
a launch it stops by force, a cold Planner on Ctrl-C included, so each
left `nvx-sandbox-run-<id>` behind with OpenVMM's log in it. The
connector now reads the folder from the OpenVMM command line `nvx`
prints, records it with the launch and removes it when the launch ends,
however it ends. A dead connector's records take their folders with
them, and `chopin connect` sweeps those no launch record or running
process names (OpenVMM's command line does for its whole run) and that
are older than a minute. It looks where `nvx` makes them: `TMPDIR` on
Linux, `GetTempPath` on Windows.

The connector's own files leave the shared temporary directory too: the
copy of the person's git configuration is in `<base>/git/<pid>`, which
the next connect removes once that process is gone, and the agent's
start check runs in the empty `<base>/nvx/check`.

Under heavy load OpenVMM sometimes can't set up io_uring for a guest's
disk and fails to start with "driver does not support io-uring" in its
log, where `nvx` says only `vmm-failure status 1`. A launch now starts
its microVM again for that failure alone, three starts at most, after
half a second and then a second and a half. The guest's launcher prints
a line as it starts the workload, and only a start that never printed it
is tried again: until then no stdin reaches the guest and nothing `nvx`
says on stderr reaches the caller, so an agent never sees two starts. On
Linux the wait is in the next start's own process group, so a stop
meanwhile ends it. A start that fails for good says why with the end of
OpenVMM's log, which the connector opens as soon as `nvx` names it.

The real-KVM tests make the failure with strace, refusing
`io_uring_setup`, which gives OpenVMM's own log line.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
@lavaman131
lavaman131 merged commit 2c2fa39 into main Nov 21, 2026
26 checks passed
@lavaman131
lavaman131 deleted the nvx-cleanup-retry branch November 21, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant