Skip to content

[Packaging] Resolve every package from one verified build#1073

Merged
brandonpayton merged 21 commits into
mainfrom
build/atomic-multi-output-current-after-1070-qk044
Jul 24, 2026
Merged

[Packaging] Resolve every package from one verified build#1073
brandonpayton merged 21 commits into
mainfrom
build/atomic-multi-output-current-after-1070-qk044

Conversation

@brandonpayton

@brandonpayton brandonpayton commented Jul 23, 2026

Copy link
Copy Markdown
Member

Why

Some Kandelo packages contain several files that must come from the same build. Python, for example, needs both an executable and runtime data. Mixing those files across builds can produce subtle startup failures.

This PR makes one package build an indivisible generation. Consumers receive the complete, verified generation or a clear error; they never assemble a package from unrelated files.

What changes

  • Resolve every executable and runtime-data file from one complete local, downloaded, or installed package generation.
  • Reject partial packages, mixed generations, stale metadata, malformed paths, undeclared files, and obsolete layouts.
  • Return immutable generation-member paths, so publishing a newer build cannot change bytes that a running consumer already selected.
  • Generate one Rust-owned program-packages.json contract shared by Rust, Node.js, browser tooling, shell scripts, external registries, and the installed host package.
  • Recheck that contract against package recipes, build revisions, toolchain inputs, registry order, and transitive dependencies.
  • Publish package indexes and mirror links transactionally, so readers cannot observe a half-written update.
  • Give the browser development server access only to exact files approved by the package resolver.
  • Install the candidate Homebrew shell through the same generation path used by normal package consumers.

Portable CI workspaces

Staging uncovered a real handoff bug: an archive step flattened trusted package symlinks into ordinary files and erased their generation identity.

The fix now:

  • copies only the content-addressed generations used by the test workspace;
  • recreates package mirrors as portable relative links;
  • materializes non-package artifacts such as the kernel as regular files; and
  • rejects dangling, absolute, escaping, flattened, or noncanonical package links before upload.

The same transport path is used by staging, prepare-merge, and force-rebuild workflows.

Browser package safety

The browser discovers its full authored package graph once and verifies that graph as a batch. This avoids repeating an expensive source check for every import while preserving package-generation integrity.

  • Vite's classification-only dependency scan does not invoke the package checker.
  • A browser session with no package bytes does not need the Rust toolchain.
  • Relative mirror imports and @binaries/... imports use the same canonical resolver.
  • A late-installed declared package refreshes the complete graph, not one member at a time.
  • Capability approval validates every member before publishing any member.
  • A failed batch remains closed and can retry after the filesystem is repaired.
  • Truly undeclared requests still use the canonical scalar resolver, which pins the requested package's complete closure.

The composite nginx-vfs, nginx-php-vfs, and redis-vfs images now also own proper package manifests instead of using anonymous package-shaped mirror paths.

Validation

Current exact base: a112375f50ecb967c8d9a23e9627f343235b78bd
Current exact head: 7382ec93b61d0891e4543114393d062ab2544d7a

Local validation on the current tree:

  • 88/88 focused package resolver, browser graph, capability, and ownership tests passed.
  • 4/4 Chromium cache-boundary tests passed with the exact prepared-checker path. Coverage proves direct checker reuse, fail-closed behavior, a clean dependency scan that does not invoke the checker, approved bottle serving, cache isolation, and explicit overlapping-cache handling.
  • Host TypeScript declaration generation passed.
  • Prepared-workspace checker selection passed, including relocated exact-path selection and fail-closed behavior when the transported checker is missing or non-executable.
  • The exact downloaded Linux staging workspace reproduced the implicit-checker failure after 31.116 seconds; selecting its packed checker made asctime/1-1 pass in 2.34 seconds with zero timeouts.
  • Program-index generation/context checks and git diff --check passed.
  • Independent review found no remaining blocker in the transactional browser repair.

Exact GitHub validation:

This PR will not merge until those exact-head workflows and prepare-merge are green.

Not activated here

This is packaging infrastructure for the remaining Homebrew migration. It does not switch the main shell image, add brew inside the guest, retire packages/registry, change the kernel or ABI, or implement bottle-declared composable VFS packages.

@brandonpayton brandonpayton changed the title [Packaging] Publish and resolve each package as one verified generation [Packaging] Resolve each package from one verified build Jul 23, 2026
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Phase B-1 matrix build status — pr-1073-staging

ABI v41. 11 built, 0 failed, 11 total.

Package Arch Status Sha
kandelo-sdk wasm32 built 58303f53
mariadb-test wasm32 built 4aa87217
mariadb-vfs wasm32 built 54750c03
mariadb-vfs wasm64 built e4d4ac50
redis-vfs wasm32 built 527cb563
shell wasm32 built 8f31a830
lamp wasm32 built c613db2f
nginx-php-vfs wasm32 built 5e700750
nginx-vfs wasm32 built 1538a047
node-vfs wasm32 built 2cbc8fc1
wordpress wasm32 built bcd426e3

Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.

@brandonpayton brandonpayton changed the title [Packaging] Resolve each package from one verified build [Packaging] Resolve every package from one verified build Jul 23, 2026
@brandonpayton brandonpayton added ready-to-ship Maintainer attests the exact tested head and requests merge preparation. and removed ready-to-ship Maintainer attests the exact tested head and requests merge preparation. labels Jul 23, 2026
@brandonpayton

Copy link
Copy Markdown
Member Author

Exact staging follow-up (2d8108b62)

The previous exact staging head built every selected package successfully. Its Vitest shard then exposed a scaling bug in the example runner: importing the command table resolved about 35 optional programs separately, so each path started the canonical Rust source-projection check and valid subprocess tests exceeded Vitest’s five-second limit.

This follow-up keeps the safety boundary and removes the repeated work:

  • one batch API checks the source projection once, then independently resolves each optional program;
  • every package member still resolves through its complete verified generation, and partial multi-file packages still fail closed;
  • the example runner describes package-backed commands lazily and does not inspect package state before usage or credential validation; and
  • tests cover one-check batching, missing optional programs, incomplete multi-member closure rejection, and both early-return paths against a deliberately mismatched cache.

Validation on the exact transported staging workspace: 78/78 focused Vitest tests passed. Host typechecking, the standalone resolver bundle build/test, program-index-check, program-index-context-check, and git diff --check also passed. Fresh full staging is running. Final prepare-merge activation is intentionally held until #1078 lands so the synthetic merge base is not made stale.

A package can contain several executable outputs, runtime data files, or both. Publishing those paths one at a time can expose a mixture of builds, and a direct local copy can accidentally follow a fetched mirror symlink into the shared cache.

Validate the complete declared closure before changing its live package directory, stage fetched mirrors beside the destination, and replace the directory as one transaction. Collect direct local builds into create-once session generations with one-shot publication claims, while retaining safe atomic replacement for one-member packages and legacy aliases. Treat outputs plus runtime files as one package identity so executable-plus-runtime packages such as CPython and Erlang use the same contract.
A complete package path could previously mix files from different builds or retarget after validation. Malformed and legacy spellings could bypass the package closure, while the browser development server exposed the entire program cache instead of only the files selected by the resolver.

Project the registry into one closed, cached package model; resolve outputs and runtime files as one verified generation; return canonical member paths; and preserve truthful not-found versus invalid-state errors. Narrow Vite serving to exact resolver-approved regular files so Node.js and browser hosts consume the same package generation without exposing neighboring cache content.
Install the reviewed shell VFS with install-local-artifact under a run-unique generation session, then resolve and compare the canonical installed bytes before browser validation. Strengthen the shell workflow contract so direct local-binaries writes cannot return.
Declare nginx, nginx+PHP, and Redis composite images as complete package generations with exact dependencies and cache inputs. This prevents their browser mirrors from bypassing the strict package-generation contract.

Add a source-level ownership audit for every composite VFS wrapper so a future image cannot call the package installer without a manifest, build sidecar, exact source artifact, canonical mirror path, and explicit non-Wasm policy.
Declare the shell-owned inputs used by nginx-derived images and narrow the nginx-only and Redis host inputs to the resolver/VFS boundaries they actually consume. Keep nginx+PHP broad because opcache prewarming boots the complete host runtime.

Extend the cache-domain test with an inline explanation of that distinction so future maintainers cannot silently over- or under-invalidate these images.
Build the synthetic package projection through xtask so the boundary test exercises the same manifest, cache-key, and source-freshness contract as production. Keep generated imports outside source-scanned paths and assemble the authored fixture string so interrupted tests and static scanners cannot create false package dependencies.
Keep package generation checks fail-closed while avoiding one canonical source-projection subprocess per optional command. Delay the example runner's package probes until after CLI input validation, and cover the independent-batch and complete-closure boundaries.
Regenerate the authoritative package projection on the exact post-#1078 base so composite VFS consumers use cache identities derived from the current source closure.
@brandonpayton
brandonpayton force-pushed the build/atomic-multi-output-current-after-1070-qk044 branch from 2d8108b to 099828f Compare July 23, 2026 18:34
@brandonpayton brandonpayton added batched-changes Use rebase auto-merge in prepare-merge so granular commits land on main. ready-to-ship Maintainer attests the exact tested head and requests merge preparation. labels Jul 24, 2026
@github-actions

Copy link
Copy Markdown

prepare-merge: test-gate passed against the synthetic PR merge and sealed merge-candidate-abi-v41-pr-1073-run-30061996531-attempt-1. The canonical ABI index is unchanged. merge-gate=success was posted on PR HEAD. This PR is ready for a reviewer to rebase merge; default-branch reconciliation will verify the exact merged tree before activation.

@brandonpayton
brandonpayton merged commit 3344994 into main Jul 24, 2026
139 of 141 checks passed
@brandonpayton
brandonpayton deleted the build/atomic-multi-output-current-after-1070-qk044 branch July 24, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

batched-changes Use rebase auto-merge in prepare-merge so granular commits land on main. ready-to-ship Maintainer attests the exact tested head and requests merge preparation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant