VFS: Verify lazy downloads before running programs#1024
Draft
brandonpayton wants to merge 1 commit into
Draft
Conversation
Phase B-1 matrix build status —
|
| Package | Arch | Status | Sha |
|---|---|---|---|
| kandelo-sdk | wasm32 | built | b0e75273 |
| rootfs | wasm32 | built | 93cf7c62 |
| shell | wasm32 | built | a73d0e0c |
| lamp | wasm32 | built | 089ad416 |
| node-vfs | wasm32 | built | bf36f02d |
| wordpress | wasm32 | built | 3e02f290 |
Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.
This was referenced Jul 20, 2026
brandonpayton
added a commit
that referenced
this pull request
Jul 21, 2026
…1037) ## Why The shell image records lazy ZIP metadata for Vim and NetHack. It previously rebuilt those ZIPs even though Kandelo already publishes `vim-browser-bundle` and `nethack-browser-bundle` outputs. Separate ZIP builds can contain the same files but still produce different bytes because path order, timestamps, permissions, environment options, or ZIP metadata differ. That is what the new lazy-archive integrity check in #1024 exposed: the shell recorded one Vim digest while Chromium fetched a different, independently generated Vim ZIP. The integrity check correctly rejected the mismatch. ## What changes ### Use one declared producer - Make `shell` depend on the Vim and NetHack browser-bundle packages instead of the underlying program packages. - Resolve the exact declared `programs/wasm32/vim.zip` and `nethack.zip` outputs. The shell no longer rebuilds either archive or falls back to a public-directory copy. - Read each output once, validate its central directory and required executable, and derive its byte length and SHA-256 from those exact bytes. ### Make independent bundle builds byte-reproducible - Add one shared deterministic ZIP helper used by both bundle builders. - Build in a private mirror so source trees are not modified. - Sort entry paths bytewise and normalize file modes, link modes, timestamps, compression level, locale, timezone, and host-specific ZIP fields. - Preserve package-owned symlinks and validate their targets before VFS registration. - Declare every byte-producing helper as a bundle cache-key input. - Keep the in-flight Vim bundle at revision 3, bump NetHack from revision 3 to 4, and keep the shell bump from revision 13 to 14. ### Protect the lazy-archive mount boundary - Preflight every member before creating directories, stubs, symlinks, inode mappings, or group metadata. - Reject absolute, traversal, backslash, non-canonical, duplicate, colliding, and non-directory-ancestor member paths. - Validate mount prefixes and required symlink targets in the same preflight. Missing, corrupt, wrong, non-reproducible, or path-escaping bundle outputs now fail at the producer/consumer contract instead of being hidden by a lookalike rebuild or partially mutating the VFS. This is a packaging, host-VFS, and VFS build change; it does not change the kernel or weaken #1024. ## Validation - Focused bundle coverage: 10 tests passed for both bundle specs, exact bytes/digests, deterministic ZIP bytes, canonical entry order, permissions, local and central timestamps/extra fields, DEFLATE bytes, symlink preservation and VFS registration, resolver calls, fail-loudly stat and newline boundaries, missing/corrupt/wrong outputs, no fallback, and package declarations. - Lazy-archive API coverage: 37 tests passed, including regular-file and symlink traversal proofs plus absolute, backslash, empty-component, duplicate, and non-directory-ancestor rejection before any VFS/group mutation. - The deterministic fixture produces canonical SHA-256 `67b198037eb2b47e6d5acadbdfc94f25ec83c469c5374d719342392b08ef0bd0` from source trees with different creation order, mtimes, permission bits, timezone, `SOURCE_DATE_EPOCH`, and `ZIPOPT`. It passed with both the Darwin and GNU command sets in the repository dev shell and asserts that the large fixture member uses DEFLATE. - Strict TypeScript compilation passed for the host VFS, lazy-archive helper, shell builder, and both focused test files. - `bash -n` passed for the shared ZIP helper and both bundle builders. - Package parsing and cache-key calculation passed in the dev shell: - `vim-browser-bundle`: `cf5f9141bfc5047f7548121cafb993708a8da25b2639967cc603e4d0347a4347` - `nethack-browser-bundle`: `c50cb85a33c8e4dd8157998a3c7fde468d33193054b7f0ce6cef111ccb26b011` - `shell`: `802b871c980a8d8e44b28ab25a0a6947e6b350f93e2b4825ad4345553f5ede7d` - Declared output-path checks resolve `vim.zip` and `nethack.zip`. - `git diff --check` passed. The full repository test path is left to CI because a fresh isolated checkout does not have the prebuilt sysroot needed by the unrelated global C-fixture setup. ## #1024 handoff Current main does not yet accept integrity metadata in `registerLazyArchiveFromEntries`, so this PR returns the exact computed integrity without pretending an ignored argument is enforced. After this producer fix lands, #1024 must rebase onto it, pass `archive.integrity` into the real integrity-aware registration API, and rerun the exact Chromium proof. Both halves are required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Kandelo is moving toward a main shell that can fetch Homebrew-provided Perl,
Python, and Erlang closures only when they are first used. A lazy executable
must not turn an HTTP 200 response, a truncated file, or a changed ZIP into
guest-visible program bytes.
The old lazy path checked neither the complete response length nor archive
identity. Node also returned lazy executable bytes directly instead of
materializing the kernel-owned VFS, so Node and browser state diverged.
What changes
at 1 GiB, and leave its stub retryable after a short, long, or failed fetch.
lowercase SHA-256. Verify both before parsing or writing members.
archive stub, bound deflate expansion, and share concurrent materialization
work per inode or archive.
that browser exec uses.
ignoring live metadata made stale by a concurrent guest unlink or write.
require exact sizes in mkrootfs manifests, and remove benchmark size guesses.
Relationship to the Homebrew work
This is the consumer-side complement to draft #1023. That PR publishes a
deterministic browser-fetchable Homebrew closure and descriptor; this PR gives
the runtime a bounded, digest-checked way to consume such a closure. The two
PRs do not edit the same files.
A follow-up can wire the #1023 descriptor into the main shell for lazy Perl,
Python, and Erlang. This PR does not add those runtime entries by itself.
This is a clean successor to stale #897, rebuilt on current main rather than a
merge of its old ABI-era branch.
Compatibility and ABI
The archive byte-count and digest fields are optional when reading existing VFS
images, so older images remain loadable. Current builders provide both fields
for newly produced remote ZIPs.
No kernel ABI shape or syscall behavior changes. ABI version 41 and the
committed snapshot remain unchanged.
Validation
passed.
Firefox, and WebKit.
The broader default host run reached 1,227 passing tests. Its remaining 34
failures were fixture setup failures because this isolated worktree does not
contain sysroot64, rootfs.vfs, or the full generated program-binary set; no
touched lazy/VFS test remained failing.