Repository navigation
Leave no nvx folders behind, and retry a microVM OpenVMM couldn't start - #978
Merged
Merged
Conversation
`nvx sandbox run` keeps a folder in the temporary directory for each run and removes it only when its workload ends by itself. The connector ends a launch it stops by force, a cold Planner on Ctrl-C included, so each left `nvx-sandbox-run-<id>` behind with OpenVMM's log in it. The connector now reads the folder from the OpenVMM command line `nvx` prints, records it with the launch and removes it when the launch ends, however it ends. A dead connector's records take their folders with them, and `chopin connect` sweeps those no launch record or running process names (OpenVMM's command line does for its whole run) and that are older than a minute. It looks where `nvx` makes them: `TMPDIR` on Linux, `GetTempPath` on Windows. The connector's own files leave the shared temporary directory too: the copy of the person's git configuration is in `<base>/git/<pid>`, which the next connect removes once that process is gone, and the agent's start check runs in the empty `<base>/nvx/check`. Under heavy load OpenVMM sometimes can't set up io_uring for a guest's disk and fails to start with "driver does not support io-uring" in its log, where `nvx` says only `vmm-failure status 1`. A launch now starts its microVM again for that failure alone, three starts at most, after half a second and then a second and a half. The guest's launcher prints a line as it starts the workload, and only a start that never printed it is tried again: until then no stdin reaches the guest and nothing `nvx` says on stderr reaches the caller, so an agent never sees two starts. On Linux the wait is in the next start's own process group, so a stop meanwhile ends it. A start that fails for good says why with the end of OpenVMM's log, which the connector opens as soon as `nvx` names it. The real-KVM tests make the failure with strace, refusing `io_uring_setup`, which gives OpenVMM's own log line. Co-authored-by: Alex Lavaee <lavaman131@github.com>
lavaman131
force-pushed
the
nvx-cleanup-retry
branch
from
November 21, 2026 13:04
88aa64b to
4ad0f94
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #975. It does the two things #975 listed as not done:
nvx sandbox run's temporary folders are no longer left behind, and a microVM start that OpenVMM fails under heavy load is tried again.Leftover folders
nvx sandbox runkeeps a private folder in the temporary directory for each run (nvx-sandbox-run-<id>). It removes the folder itself only when the workload ends on its own. The connector ends a launch it stops by force (a cold Planner on Ctrl-C, for example), so each such stop left a folder holding OpenVMM's log. This happens on Linux and on Windows.nvx sandbox runwon't put the folder anywhere else: it rejects--state-dir. PointingTMPDIRinto the state directory doesn't work either, because OpenVMM's control socket path would go past Linux's 108-byte limit. So the connector now:nvxprints (--microvm-report <folder>/outcome.json,runFolderOf) and records it in the launch'slaunch.json;endOrphans);chopin connectstarts, removes the leftovers no launch record names (sweepRunFolders). It looks wherenvxcreates them:$TMPDIRor/tmpon Linux (not$TMP), andGetTempPathon Windows. It keeps a folder that belongs to another user, is less than a minute old, or is named by a running process. OpenVMM's command line names its folder for the whole run, so this covers a microVM of another chopin on another base dir, or any other NVX user's. The connector reads command lines from/procon Linux and fromWin32_Processon Windows; if it can't read them, it keeps every folder.The connector's own files are also out of the shared temporary directory:
chopin-git-*) is now in<base>/git/<pid>/. It is removed when that process exits, andchopin connectremoves any folder whose process is gone (sweepGitFolders).chopin-check-*, or<base>/check-*forservice install) now runs in one empty folder,<base>/nvx/check.Starting a microVM again
Under heavy load OpenVMM sometimes fails at startup. Its log ends:
nvxreports onlyvmm-failure status 1. The cause isio_uring_setupfailing for OpenVMM's block driver. Refusing that one syscall withstrace -e inject=io_uring_setup:error=ENOMEMproduces exactly this log, so the real-KVM tests use it to cause the failure.startVmnow returns one launch that may startnvxup to three times. It waits 0.5 s before the second start and 1.5 s before the third. It retries only when both of these hold:nvxprints the folder:nvxcreates the log before printing the line and deletes it when a start fails, and an open file descriptor keeps it readable;>> chopin: the workload startson stdout just before itexecs the agent (STARTED_LINE). Until that line arrives, the launch sends nothing of its stdin to the guest and passes nothing ofnvx's stderr to the caller. Once it arrives, the connector strips it, as it already stripsnvx's own launch line. So an agent never sees two starts, and never gets input written to a start that failed.A stop during the wait is honored:
sleepinside the next start's own process group. The caller's group signal ends that group, and the launch ends without starting again.stopBackendre-reads the group each time it polls, in case a start fails just as it is stopped.stopMicrovm, which cancels the wait.A start that fails for good, either after three tries or at once for any other reason, writes one stderr line with the end of OpenVMM's log. That line is what a run or Planner failure shows:
chopin connectreports each new start in its output. The CI microVM job installsstraceif the runner lacks it.Invariants and their tests
nvx's in the temporary directory, however it endssandbox.test.ts: a launch ended by force leaves no folder, and a dead connector's record takes its folder.nvx-vm.test.ts(KVM): a launch SIGKILLed while running leaves neither its recorded folder nor its recordsandbox.test.ts: a folder survives a sweep by another connector on the same base dir (it is in a record) and on another base dir (a process names it); a young folder survives; a stale folder is removedsandbox.test.tswith a fakenvx: one io-uring failure then a start reads its stdin once; another failure is not retried; a start whose workload had started is not retried; three starts at most; a stop during the wait starts nothing.nvx-vm.test.ts(KVM, strace): the workload runs and reads its input oncesandbox.test.ts(KVM log and io-uring log);nvx-vm.test.ts(three real failures)git-host.test.ts(the copy is in<base>/git/<pid>, and the sweep removes only folders of dead processes);main.test.ts(the start check runs in<base>/nvx/check)Each new behaviour test failed when its guard was removed: without the stdin gate, without the removal on end, without the process check, and without both started guards.
Verification
bun run fix,bun run ci,bun run typesand the connector unit tests ran on the final head4ad0f949(on6f7a699e). The KVM tests and the connector-related browser tests ran on88aa64bc, the same change onfca5f0b0. The Windows checks and the first browser run were made before the rebase ontofca5f0b0:bun run fixmade no changes.bun run ciandbun run typespassed.bun test apps/connector: 431 pass, 29 skip, 0 fail.CHOPIN_NVX_ROOT=<nvx-0.1.0-linux-kvm> bun test apps/connector/src/nvx-vm.test.ts: 12 pass, including the three new tests.startVmand sweep with the realnvx.exe:kill()remove the folder. While a VM runs, its folder holds onlyopenvmm.log. The control channel there is a named pipe, so the folder appears only in--microvm-report.Win32_Process(0.76 s) removes a stale folder and keeps one a process names and a young one. It also removed the six real leftovers from earlier runs in%TEMP%.The microVM didn't start: fatal error: failed to launch vm worker / … / The paging file is too small for this operation to complete. (os error 1455) (nvx: …)and is not retried.flockwith their own port base and PostgreSQL container:no-ai-account, and two of them fail identically onorigin/mainwithout this change.88aa64bcwithout those two files: 144 passed, 0 failed.