Skip to content

Homebrew VFS: Reuse declared lazy-bundle outputs for shell archives#1037

Merged
brandonpayton merged 3 commits into
mainfrom
fix/shell-use-declared-lazy-bundles
Jul 21, 2026
Merged

Homebrew VFS: Reuse declared lazy-bundle outputs for shell archives#1037
brandonpayton merged 3 commits into
mainfrom
fix/shell-use-declared-lazy-bundles

Conversation

@brandonpayton

@brandonpayton brandonpayton commented Jul 21, 2026

Copy link
Copy Markdown
Member

Why

The shell image records lazy ZIP metadata for Vim and NetHack. It previously
rebuilt those ZIPs even though Kandelo already publishes
vim-browser-bundle and nethack-browser-bundle outputs. Separate ZIP builds
can contain the same files but still produce different bytes because path
order, timestamps, permissions, environment options, or ZIP metadata differ.

That is what the new lazy-archive integrity check in #1024 exposed: the shell
recorded one Vim digest while Chromium fetched a different, independently
generated Vim ZIP. The integrity check correctly rejected the mismatch.

What changes

Use one declared producer

  • Make shell depend on the Vim and NetHack browser-bundle packages instead
    of the underlying program packages.
  • Resolve the exact declared programs/wasm32/vim.zip and nethack.zip
    outputs. The shell no longer rebuilds either archive or falls back to a
    public-directory copy.
  • Read each output once, validate its central directory and required
    executable, and derive its byte length and SHA-256 from those exact bytes.

Make independent bundle builds byte-reproducible

  • Add one shared deterministic ZIP helper used by both bundle builders.
  • Build in a private mirror so source trees are not modified.
  • Sort entry paths bytewise and normalize file modes, link modes, timestamps,
    compression level, locale, timezone, and host-specific ZIP fields.
  • Preserve package-owned symlinks and validate their targets before VFS
    registration.
  • Declare every byte-producing helper as a bundle cache-key input.
  • Keep the in-flight Vim bundle at revision 3, bump NetHack from revision 3 to
    4, and keep the shell bump from revision 13 to 14.

Protect the lazy-archive mount boundary

  • Preflight every member before creating directories, stubs, symlinks, inode
    mappings, or group metadata.
  • Reject absolute, traversal, backslash, non-canonical, duplicate, colliding,
    and non-directory-ancestor member paths.
  • Validate mount prefixes and required symlink targets in the same preflight.

Missing, corrupt, wrong, non-reproducible, or path-escaping bundle outputs now
fail at the producer/consumer contract instead of being hidden by a lookalike
rebuild or partially mutating the VFS. This is a packaging, host-VFS, and VFS
build change; it does not change the kernel or weaken #1024.

Validation

  • Focused bundle coverage: 10 tests passed for both bundle specs, exact
    bytes/digests, deterministic ZIP bytes, canonical entry order, permissions,
    local and central timestamps/extra fields, DEFLATE bytes, symlink
    preservation and VFS registration, resolver calls, fail-loudly stat and
    newline boundaries, missing/corrupt/wrong outputs, no fallback, and package
    declarations.
  • Lazy-archive API coverage: 37 tests passed, including regular-file and
    symlink traversal proofs plus absolute, backslash, empty-component,
    duplicate, and non-directory-ancestor rejection before any VFS/group
    mutation.
  • The deterministic fixture produces canonical SHA-256
    67b198037eb2b47e6d5acadbdfc94f25ec83c469c5374d719342392b08ef0bd0
    from source trees with different creation order, mtimes, permission bits,
    timezone, SOURCE_DATE_EPOCH, and ZIPOPT. It passed with both the Darwin
    and GNU command sets in the repository dev shell and asserts that the large
    fixture member uses DEFLATE.
  • Strict TypeScript compilation passed for the host VFS, lazy-archive helper,
    shell builder, and both focused test files.
  • bash -n passed for the shared ZIP helper and both bundle builders.
  • Package parsing and cache-key calculation passed in the dev shell:
    • vim-browser-bundle: cf5f9141bfc5047f7548121cafb993708a8da25b2639967cc603e4d0347a4347
    • nethack-browser-bundle: c50cb85a33c8e4dd8157998a3c7fde468d33193054b7f0ce6cef111ccb26b011
    • shell: 802b871c980a8d8e44b28ab25a0a6947e6b350f93e2b4825ad4345553f5ede7d
  • Declared output-path checks resolve vim.zip and nethack.zip.
  • git diff --check passed.

The full repository test path is left to CI because a fresh isolated checkout
does not have the prebuilt sysroot needed by the unrelated global C-fixture
setup.

#1024 handoff

Current main does not yet accept integrity metadata in
registerLazyArchiveFromEntries, so this PR returns the exact computed
integrity without pretending an ignored argument is enforced. After this
producer fix lands, #1024 must rebase onto it, pass archive.integrity into
the real integrity-aware registration API, and rerun the exact Chromium proof.
Both halves are required.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

Phase B-1 matrix build status — pr-1037-staging

ABI v41. 8 built, 0 failed, 8 total.

Package Arch Status Sha
kandelo-sdk wasm32 built fc1d0809
nethack-browser-bundle wasm32 built c50cb85a
rootfs wasm32 built b1f4c0f2
vim-browser-bundle wasm32 built cf5f9141
shell wasm32 built 3b6ac0b5
lamp wasm32 built 01a260b1
node-vfs wasm32 built f3751905
wordpress wasm32 built d2d574b8

Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.

@brandonpayton
brandonpayton force-pushed the fix/shell-use-declared-lazy-bundles branch 2 times, most recently from 0448037 to 56cd0e0 Compare July 21, 2026 09:03
Canonicalize archive order, timestamps, modes, compression metadata, and symlink storage so independent producer and image jobs share one byte identity. Cover the canonical bytes and VFS registration contract with a cross-tool fixture.
@brandonpayton
brandonpayton force-pushed the fix/shell-use-declared-lazy-bundles branch from 56cd0e0 to afcd937 Compare July 21, 2026 11:51
@brandonpayton brandonpayton added the ready-to-ship Maintainer attests the exact tested head and requests merge preparation. label Jul 21, 2026
@github-actions

Copy link
Copy Markdown

prepare-merge: test-gate passed against the synthetic PR merge and sealed merge-candidate-abi-v41-pr-1037-run-29830755038-attempt-1. The canonical ABI index is unchanged. merge-gate=success was posted on PR HEAD. This PR is ready for a reviewer to squash merge; default-branch reconciliation will verify the exact merged tree before activation.

@brandonpayton
brandonpayton merged commit ae8d4ec into main Jul 21, 2026
74 checks passed
@brandonpayton
brandonpayton deleted the fix/shell-use-declared-lazy-bundles branch July 21, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-ship Maintainer attests the exact tested head and requests merge preparation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant