libxpkg 0.0.52: host_link_interposer, and a mixed deps table that dropped its build deps - #36
Merged
Merged
Conversation
…ped to one object
A driver vendor library is the HOST's file: a symlink into /usr/lib, coupled to
the host's kernel module, and not ours to put an RPATH on. Its DT_NEEDED
entries therefore have three fates when it is dlopen'd into one of our
processes — already loaded (fine), findable as the host's copy (two builds of
one library in one process), or not findable at all (the vendor fails, no GPU).
Our loader's built-in search path cannot exist by construction (AD-5), so the
third is the default. `DEVICE_COUNT=0`. The historical answer was to put our
libraries on LD_LIBRARY_PATH, which reaches the first two AND hands them to
every other process in the subos, including host binaries on the host loader.
That is how `xlings subos use` once returned a /bin/bash that died of SIGSEGV
before printing a character.
This does the same job with a scope of exactly one object: a tiny shared
library that takes the vendor's SONAME, NEEDs the real vendor by absolute path
so dlsym still reaches its entry points through the handle's dependency tree,
and carries DT_RPATH — not RUNPATH — naming the payload closure, because
DT_RPATH is transitive along the load chain and RUNPATH is not. Nothing lands
on any process-global variable.
Measured on a real NVIDIA stack, not argued:
subos today (LD_LIBRARY_PATH + lib/xlings-deps) llvmpipe
interposer, LD_LIBRARY_PATH = host driver dir only RTX 4080, RESULT=ok
The probe renders and reads the pixel back, because a renderer string printing
is something a half-working stack does too.
`libdirs` defaults to closure_lib_paths(), which is the resolver's own answer.
Passing a hand-written list is possible and is a mistake: the table this
replaces was hand-written and missing libm, libdrm, libgbm, libgcc_s and
libwayland-*, every one of which was silently coming from the host (R7).
PRECONDITION, measured in the same experiment and written into the contract:
an object produced here may only be loaded by a consumer whose INTERP points
into our payload. Handing one to a host binary fails as `librt.so.1: undefined
symbol: __pointer_chk_guard, version GLIBC_PRIVATE` — the loader/libc split,
from the direction the same-source assertion cannot see.
The result is asserted, not assumed (R4). Each of the three checks exists
because its absence yields an interposer that loads perfectly and does nothing:
a wrong SONAME is never asked for; a missing NEEDED leaves dlsym with no entry
point, so the caller reports "no device" rather than an error; DT_RUNPATH is
not transitive, so the vendor's dependencies fall through to the host. An empty
closure and a missing vendor are refused for the same reason.
Version 0.0.52. Three tests.
Three shapes reach the deps parser:
deps = { "a", "b" } array -> fan out to both
deps = { runtime = {...}, build = {...} } split -> separate lists
deps = { "a", "b", build = {...} } mixed -> was silently wrong
The third is what an author writes when they already have a list of
runtime deps and want to add one install-time tool. It parses, it reads
exactly like the split form, and the loader took the array branch on it:
`build` was dropped on the floor, and the ARRAY entries were copied into
build_deps in its place.
Measured on nvidia-gl-host-link (2026-08-06): `deps.build =
{"xim:patchelf@0.18.0"}` alongside five runtime deps installed no
patchelf at all, and the install reported success. The tool that BUILDS
the interposer then resolved to the host's, which elfpatch warned about
-- so the symptom surfaced two layers away from the cause, as a warning
about a different subsystem.
Now an author who writes `runtime` or `build` at all has opted into the
split, so both keys are honoured and the array part folds into runtime
-- which is the only thing a positional list beside `build = {...}` can
mean. Array-only is untouched: every existing recipe is that shape, and
its fan-out is load-bearing.
The gap was in the tests too. Both pure shapes were covered; the mixed
one was not, and it is the one an author arrives at by editing an
existing recipe rather than writing a new one. The new test asserts
BOTH halves -- checking only that build_deps holds patchelf would still
pass if the array were also being fanned into it.
Sunrisepeak
added a commit
to openxlings/xlings
that referenced
this pull request
Aug 6, 2026
…oven by provenance (#490) * docs: the B line's core claim, proven on the real NVIDIA stack The gate (§2.7) proved the MECHANISM works — DT_RPATH transitivity, dlsym through the handle, GLX needing no process-global variable. This proves it works on the actual driver, and measures the boundary the gate could not see. Baseline, with glprobe, which renders and reads a pixel back rather than printing a version string: host env GL_RENDERER = NVIDIA GeForce RTX 4080/PCIe/SSE2 PIXEL=336699 in subos GL_RENDERER = llvmpipe (LLVM 20.1.7, 256 bits) PIXEL=336699 §2.6's defect, measured: the same host-linked binary drops from the GPU to software rendering inside the subos, silently. Both pixels are correct, so "does it render" cannot catch this — only the renderer name can. With a 27 KB interposer built by patchelf alone, and LD_LIBRARY_PATH carrying ONLY the host driver directory — lib/xlings-deps not on it at all: GL_RENDERER = NVIDIA GeForce RTX 4080/PCIe/SSE2 PIXEL=336699 RESULT=ok B2's acceptance criterion, satisfied. The boundary came out of the same experiment. Handing that interposer to a HOST binary fails as librt.so.1: undefined symbol: __pointer_chk_guard, version GLIBC_PRIVATE which is the 2026-08-05 crash verbatim: the interposer's RPATH names OUR glibc, and the consumer's libc is the host's. Not a defect — the domain of applicability, and consistent with §2.3, which says the process the vendor is dlopen'd into is ours by construction. But it is a precondition B1 has to put in the contract rather than leave implied: an object produced by host_link_interposer may only be loaded by a consumer whose INTERP points into our payload; host binaries must keep using the host's own vendor. Which is precisely the argument for B4: bake the vendor directory into the libglvnd WE build, and the two paths separate by construction instead of by an environment variable that every child inherits. Also corrects two wrong calls of mine about the install itself. It was never network-infeasible: a fresh isolated home defaults to the GLOBAL mirror, and `xlings config --mirror CN` installs all 22 packages in minutes. Before that I had declared it hung on twenty seconds of no growth in one directory, while it was between finishing its downloads and extracting them. * 2026.8.6.3: the B line lands, and a loader shape that dropped build deps Pins libxpkg 0.0.52, which brings two things: - `elfpatch.host_link_interposer` — the mechanism the B line is built on. From a shipped empty ELF stub, patchelf produces an object with the vendor's SONAME, the HOST vendor as an absolute DT_NEEDED, and the payload closure as DT_RPATH. The vendor's dependencies then resolve out of our payloads along one load chain, instead of being broadcast to the whole process through LD_LIBRARY_PATH. - A `deps` table mixing a positional list with `build = {...}` used to drop the build deps silently and copy the positional entries into build_deps in their place. Declared, reported as installed, neither done. Companion: openxlings/libxpkg#36, openxlings/xim-pkgindex#532. The doc's §7.7 records how this went, because the interesting part is that every step produced a passing result first: - an empty payload that installed "successfully" — caught only by the hook's own assertion, and caused by a resource key nested one level too deep; - `glxinfo` printing "NVIDIA GeForce RTX 4080/PCIe/SSE2" inside the subos while every object came from /usr/lib — the host binary under the host loader, our payload contributing nothing. That output is identical whether the B line works or never happened; - `interposer: yes` with one of four glvnd entry points covered. Each vendor library is dlopened BY NAME and so is the root of its own load chain; DT_RPATH is transitive only down a chain. EGL rendered on the GPU while GLX pulled its whole closure from the host. Each was found by measuring the artifact rather than the log line, and each fix is now asserted by `xim-pkgindex/.agents/tools/graphics/verify-host-link.sh` — 12 checks on a real RTX 4080, driver 550.144.03, in an isolated home. Also corrects a claim I made earlier in this work: elfpatch does NOT silently fall back to the host's patchelf. `_find_tool` resolves payload → subos view → home bin → host, warns when it leaves the payload, and `tool_payload_dir` scans the whole store — so a home that has patchelf at all uses it, declared or not. The real gap is narrow: a home that has never installed patchelf. * pin libxpkg 0.0.53: install_dir names the cause (#487) 0.0.53 adds the diagnostic half of #487. `cannot get install dir` named an internal state and covered two causes pointing in opposite directions: - not a dependency of this package ON THIS PLATFORM - declared here, but the payload never landed It now says which, and names the package, the platform, and what the deps here actually are. The issue's own hypothesis -- that dependency resolution does not filter by platform -- does not hold: resolver.cppm reads `pkg->xpm.runtime_deps.find(platform)`, and ollama declares the CUDA sentinel only under `xpm.linux`. The cause was ollama's install hook branching on `is_host("windows")` when the real distinction was linux, so macOS took the linux path. Fixed in openxlings/xim-pkgindex#532; the message is fixed here because it is what sent the reader to paths. * ci: bump mcpp to 2026.8.6.1 so it can see a freshly published index CI pinned mcpp 2026.8.3.3 while the index's latest is 2026.8.6.1, and that pin is what made `mcpplibs.xpkg@0.0.53` read as "not found" long after it was published. The index side was verified correct at every layer before touching this: the mcpp-index commit has the 0.0.53 entry, the published artifact `mcpp-index-84fa166.tar.gz` contains it, the rolling pointer `mcpp-index-pointers.json` names 84fa166, and `releases/latest` resolves to that tag. A local mcpp 2026.8.5.4 has 0.0.53 in its live index copy. So it was not publish lag and not a stale pointer. The workflows' `xlings update` step refreshes the XIM index and its three sub-indexes -- visible in the log -- and never touches the mcpplibs one, which is fetched by mcpp itself. Bumping mcpp means bumping XIM_PKGINDEX_REF with it, in all six workflows: the two are pinned as a known-good pair, and moving one alone is how a run ends up resolving a new client against an old index. The new ref is xim-pkgindex bf969a6 -- main with #532 (the B line) merged, so CI resolves against the index that carries interposer-stub. * 2026.8.6.3: the client's own version constant The version-consistency contract caught it: mcpp.toml said 2026.8.6.3 and src/core/config.cppm still said 2026.8.6.2. A release whose binary reports the previous version is exactly what that contract exists to stop. * ci: installing mcpp is not switching to it, so assert the version `xlings install -y` lays the payload down; the shim keeps resolving whatever version was already active. The client says so -- "installed, but 'x' still resolves to ..." -- but with ~/.mcpp restored from an Actions cache that one line is the only sign that the mcpp about to run is the old build. Measured 2026-08-06: `.xlings.json` pinned mcpp 2026.8.6.1, the run used the cached 2026.8.3.3, and its index snapshot predated the dependency this PR needs. It surfaced as `mcpplibs.xpkg@0.0.53 not found` against an index that demonstrably had it -- the mcpp-index commit, the published artifact mcpp-index-84fa166.tar.gz, the rolling pointer, and releases/latest all carry the entry, and a local mcpp 2026.8.5.4 refetches it from a wiped registry on both mirrors. Every layer I could check was correct, which is what made the version the last place to look. So: switch explicitly, then ASSERT the version rather than print it. A version line in a log is only ever read after something has already gone wrong; a failed assert names the cause at the point it happens. Seven workflows, including release.yml. * ci: on a build failure, print the index the runner actually holds `mcpplibs.xpkg@<ver> not found` has now been diagnosed five times from the outside. Each check was correct: index commit has the entry published artifact mcpp-index-84fa166.tar.gz has it rolling pointer names 84fa166 releases/latest resolves to that tag publish lag still failing 30+ minutes later mcpp version pin bumped 2026.8.3.3 -> 2026.8.6.1, assert passes install-vs-use the assert proves the right mcpp runs mcpp.lock hash regenerates byte-identical from the published index Every one of those is inference about a file only the runner can see. So the failure path now prints it: the version keys in the mcpplibs index under both registry roots, the index snapshots present, and `mcpp self version`. Guessing a sixth time would cost another CI round either way; this way the round produces an answer instead of another elimination. * ci: the diagnostic hung off a command that never ran The index dump added one commit ago sat inside `mcpp test || { ... }`, with `mcpp build` running unguarded above it. The shell is `bash -e`, so a failing `mcpp build` aborted the step before the block was reached. The diagnostic existed, never ran, and the failure output was identical to a run where it had -- an entire CI round spent adding output that could not print. Now it is a separate step with `if: failure()`, so it covers any failing step in the job rather than one command's non-zero exit, and it also reports both registry roots, the snapshots present, and the mcpp/xlings versions. * ci: force `mcpp index update`, and assert what the refresh landed The prune drops the resolved indexes, and mcpp's own re-fetch happily uses what it finds in the restored Actions cache -- so a freshly published dependency reads as "not found" against an index that demonstrably has it. `mcpplibs.xpkg@0.0.53` failed this way while the index commit, the published artifact mcpp-index-84fa166.tar.gz, the rolling pointer and releases/latest all carried the entry, and a local mcpp at the same version refetched it from a wiped registry on both mirrors. `mcpp index update` is the refresh. `mcpp index status` prints the revision each index is actually at -- locally, `mcpplibs 84fa166`, which is the one datum this failure needed and nothing was printing. The assert matters as much as the refresh: calling update alone would leave "refreshed" and "still stale" producing identical output, which is the exact failure mode the step exists to end. It greps the refreshed index for the version mcpp.toml pins and, on a miss, prints the index revision and the versions that ARE there before failing. Seven workflows, including release.yml. * ci: the index assert must slice the platform section first A whole-file grep for `["0.0.53"]` passes on a file where the entry exists only under `xpm.linux` -- which is exactly the bug this assert was written to catch and did not. Both mcpp-index bumps edited the linux section alone; linux CI went green and macOS/Windows failed against a file that literally contains the string. Falsified before committing: against the broken file the check reports linux ok, macosx MISSING, windows MISSING; against the fixed one all three pass. * ci: the Windows index assert was still a whole-file match The platform-aware fix landed in six workflows and skipped this one -- the PowerShell block did not match the POSIX text my patch keyed on. So Windows kept the version of the check that passes on the broken file: `Select-String` over the whole descriptor finds `["0.0.53"]` in the linux block and reports success. Measured on this PR: macOS failed AT the assert, naming `xpm.macosx`, while Windows sailed through it and failed two steps later in `mcpp build` with a message about a package rather than a platform -- the exact difference the assert exists to make. Now it slices the `windows = {` block first, and on a miss prints the index revision and the versions present anywhere in the file, so the "it's right there" confusion is answered in the same output. Falsified before committing by replicating the slicing rule against the pre-fix descriptor: windows-block MISSING on the broken file, ok on the fixed one. * ci: the index assert reads the pin from mcpp.toml, never a literal `0.0.53` was baked into all seven workflows. That is a value that changes: the next libxpkg bump moves `mcpp.toml` and leaves every workflow asserting the previous version, which then passes against an index that still carries it. The check would go stale without ever failing -- the same shape as everything else this session has been chasing, this time in the check itself. Now the version is read from `[dependencies.mcpplibs] xpkg` at runtime, so the only constant is where the source of truth lives. Falsified before committing, all four combinations: new index + pin 0.0.53 / macosx ok new index + pin 0.0.53 / windows ok old index + pin 0.0.53 / macosx FAIL (the platform-slice defect) new index + pin 0.0.99 / macosx FAIL (the pin moved, assert follows) The first extraction attempt returned an empty string -- `gsub(/.*"|".*/,"")` is greedy to the LAST quote -- and only the falsification caught it. It would have made every job fail on the guard rather than pass silently, but wrong either way. * ci: drop the index assert, keep the refresh Per review: `mcpp index update` alone is enough. The assert added on top of it turned a transient condition into a hard failure on a timer nobody controls, and its own literal version was a value that changes. `mcpp index status` stays -- it prints the revision each index landed on, which is what makes a stale-index failure readable instead of surfacing two layers away as `<pkg>@<ver> not found`. Seven workflows; release.yml had three copies of the block.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes, both landing the B line of the subos architecture proposal.
1.
elfpatch.host_link_interposerProduces, from a shipped empty ELF stub, an object that:
--force-rpath).Loading it therefore pulls in the host's vendor, and because DT_RPATH is
transitive down the load chain, the vendor's own dependencies resolve out
of our payloads — scoped to one load chain instead of broadcast to the
whole process through
LD_LIBRARY_PATH.patchelf only: there is no compiler at install time, and patchelf edits
objects rather than creating them. The shape is asserted on the artifact
after the fact (SONAME, NEEDED, RPATH), and the function refuses an empty
closure or a missing vendor rather than emitting an object that would fail
later at
dlopenwith nothing naming the package.2. A mixed
depstable lost its build deps, silentlyThe third shape is what an author writes when they already have a list of
runtime deps and want to add one install-time tool. The loader took the
array branch on it:
buildwas dropped, and the array entries were copiedinto
build_depsin its place.Measured on
nvidia-gl-host-link:deps.build = {"xim:patchelf@0.18.0"}beside five runtime deps installed no patchelf, and the install reported
success. The symptom surfaced two layers away — as an elfpatch warning
about resolving patchelf from the host.
Both pure shapes had tests. The mixed one did not, and it is the shape an
author arrives at by editing an existing recipe rather than writing a new
one. The new test asserts both halves: checking only that
build_depsholds patchelf would still pass if the array were also being fanned into
it.
Verification
mcpp test: 115 tests pass (52 executor + 24 index + 34 loader + 5).xim-pkgindex/.agents/tools/graphics/verify-host-link.sh— 12/12,EGL and GLX both rendering through our interposer with
LD_LIBRARY_PATHempty and the host's driver files untouched.Companion: openxlings/xim-pkgindex#TBD (interposer-stub + recipe switch).