feat: Add support for SOURCE packages by toprinse · Pull Request #244 · Quantco/pixi-pack

Thibaut OPRINSEN (toprinse) · 2025-11-24T14:27:24Z

Motivation

This PR adds support for building packages from source, a feature that is currently missing. See issue #80 for background and motivation.

When a local package is found in the pixi.lock of the main package, start a pixi-build command to build the local dependencies and add them as injected packages in the environment.tar

Changes

Introduces a pixi-build command to automatically build local packages from the main pixi.lock
Adding integration test to demonstrate the the added code is working as expected.

feat: try to add intergation tests

fix: remove unused fixture fix: fix typo fix: fix typo fix: fix typo fix: fix test on arm-64, osx and win builds fix: fix test on arm-64, osx and win builds fix: add valid checksum for other architectures fix: fix test using correct checksum fix: fix typo error fix: fix typo

fix: fix typo fix: fix typo

Pavel Zwerschke (pavelzw)

Looks very solid already, thanks!
I have a couple of comments

Pavel Zwerschke (pavelzw) · 2025-11-30T09:51:25Z

+                let pkg_dir = base_dir.join(match &source_data.location {
+                    UrlOrPath::Path(p) => p.to_path().to_string(),
+                    UrlOrPath::Url(u) => {
+                        anyhow::bail!("Unexpected URL for local package source: {}", u)


Is this a thing that will be supported eventually Hofer-Julian? Depending on other pixi projects from git is also not supported yet, right?

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

Thibaut OPRINSEN (toprinse) · 2025-12-01T12:06:15Z

Thanks Pavel Zwerschke (@pavelzw) ! So I made the requested changes, could you review the new version and tell me if there are some changes that are not relevant or things I could have done better ?

Pavel Zwerschke (pavelzw) · 2025-12-01T17:29:36Z

+        .arg(output_dir)
+        .arg("--build-platform")
+        .arg(platform)
+        .current_dir(manifest_dir);


with things like PIXI_PROJECT_ROOT (set by pixi shell) i think it's safer to explicitly use pixi build --path

also, another thing i noticed:

❯ cd examples/local-build/ ❯ cargo run --bin pixi-pack -- Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.99s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack` ⠁ Building local-build-main-pkg-0.1.0-hbf21a9e_0.conda ❌ Build failed with exit code: Some(1) Command: pixi build --output-dir /var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/.tmpZQnitN --build-platform osx-arm64 (in /Users/pavel/projects/pixi-pack/examples) 📤 stderr: Error: × the source directory '/Users/pavel/projects/pixi-pack/examples', does not contain a supported manifest help: Ensure that the source directory contains a valid pixi.toml, pyproject.toml, recipe.yaml, package.xml or mojoproject.toml file. Error: Failed to build package in /Users/pavel/projects/pixi-pack/examples ❯ cargo run --bin pixi-pack -- pixi.toml Compiling pixi-pack v0.7.5 (/Users/pavel/projects/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.78s Running `/Users/pavel/projects/pixi-pack/target/debug/pixi-pack pixi.toml` ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 14 packages... 📦 Created pack at /Users/pavel/projects/pixi-pack/examples/local-build/environment.tar with size 7.70 MiB.

something is still wrong with the manifest path...

I updated the base_dir logic to correctly handle manifest paths when building source packages. Previously, when options.manifest_path was a directory (not a file), the code incorrectly used its parent directory as the base, causing build failures with paths like /examples/./pixi.toml. Now it properly uses the manifest directory itself as the base when it's a directory. Also added canonicalize() to resolve relative paths (like ./) and ensure the manifest file exists before attempting to build.

oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling) cargo run --bin pixi-pack -- Compiling pixi-pack v0.7.5 (/home/oprinsen/pixi-pack) Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.62s Running `/home/oprinsen/pixi-pack/target/debug/pixi-pack` DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "." })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-main-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-0.1.0-hbf21a9e_0.conda DEBUG: options.manifest_path = /home/oprinsen/pixi-pack/examples/local-build DEBUG: source_data.location = Path(Unix(Utf8PathBuf { _encoding: "unix", inner: "local-pkg2" })) DEBUG: base_dir = /home/oprinsen/pixi-pack/examples/local-build ✅ Built local-build-local-pkg-2-0.1.0-hbf21a9e_0.conda ⏳ Downloading 21 packages... 📦 Created pack at /home/oprinsen/pixi-pack/examples/local-build/environment.tar with size 15.28 MiB. oprinsen@lappc-p1138 local-build ⌥ (add_source_pkg_handling)

Pavel Zwerschke (pavelzw) · 2025-12-01T17:50:08Z

pushed some adjustments to your branch; after the manifest dir fixes, this is good to merge 👍🏻

fix: fix typo

Thibaut OPRINSEN (toprinse) · 2025-12-08T16:05:53Z

Hello Pavel Zwerschke (@pavelzw), I saw that the tests have failed after merging main into my branch. Is the issue linked to the hash or the build_version of the generated packages which has changed after merging the two branches ? I don't really know how to make the test reliable, as we hardcoded the hash and version in the tests, and I don't really know how to do it in another way. If I can do anything, just let me know !

Pavel Zwerschke (pavelzw) · 2025-12-08T18:46:20Z

yeah this looks really weird to me. sounds like a bug in pixi tbh.
i will create a minimal reproducer and see what the pixi people think of this one

Thibaut OPRINSEN (toprinse) · 2025-12-18T08:19:14Z

Hello Pavel Zwerschke (@pavelzw).

So I looked into the issue of the regenerated hashes and it seems that recently the cmake build backend used in the tests has been updated and the hash generation might have been impacted. I'm not sure about how it impacted the tests and it's still unclear to me but for now I added the new generated hashes using the cmake build backend 0.3.5 (maybe it could be intersting to pin the build backend version in the test to avoid new strange behaviors).

Another idea could be to use match patterns instead of the exact version of the package containing the hashes. Instead of looking for conda-meta/local-build-main-pkg-0.1.0-hb0f4dca_0.json we could also use something like starts_with("local-build-main-pkg-0.1.0-") inside the conda-meta directory.

I also updated the pixi version to use 0.61.0 so the API of pixi build command has changed a bit in pack.rs

Pavel Zwerschke (pavelzw) · 2025-12-18T08:34:39Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

Thibaut OPRINSEN (toprinse) · 2025-12-18T09:20:00Z

Sorry for not coming back to you, lost track of this pr. One thing I'm wondering is why the build string from the pixi.lock didn't match the build string of the built package 🤔 theoretically the lockfile should always match the package exactly and thus represent the correct build string already? Maybe I'm missing something, will investigate later

In general, i think pinning the build backend makes sense in this case 👍🏻

I think it would be nice if we even persisted the build backends and host+build environments used in pixi.lock (optionally): prefix-dev/pixi#5161

No problem, I was also really busy!

I can pin the build backend version in the tests to the latest version (0.3.5).

Regarding the build string mismatch issue, I'm not sure what approach you'd prefer: should we wait for pixi to provide a clear fix for this issue, or would you prefer to merge the PR as-is and address the build string inconsistency once pixi implements the changes mentioned in the issue you linked?

Pavel Zwerschke (pavelzw) · 2025-12-18T09:40:25Z

I can pin the build backend version in the tests to the latest version (0.3.5).

sounds good, maybe also link https://github.com/prefix-dev/pixi/issues/5161 next to the pin.

Regarding the build string mismatch issue

i would like to understand the issue a bit better and at least have an issue to point to. i'll look into creating a minimal reproducer for this problem and let's see what the prefix-dev team has to say

Thibaut OPRINSEN (toprinse) · 2025-12-18T09:54:39Z

I can pin the build backend version in the tests to the latest version (0.3.5).

sounds good, maybe also link https://github.com/prefix-dev/pixi/issues/5161 next to the pin.

Regarding the build string mismatch issue

i would like to understand the issue a bit better and at least have an issue to point to. i'll look into creating a minimal reproducer for this problem and let's see what the prefix-dev team has to say

Sounds good! I will pin the version and follow the issue tracking on pixi side. I also tried to do a small reproducer but I did not manage to do something showing the exact same issue.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

regenerate lock file to use pixi 0.61.0 fix: pin the build backend issue

vincent-pollet · 2026-02-05T13:48:07Z

Hello! May I kindly ask what is the status about this PR? Anything that we could help with?

vincent-pollet · 2026-03-16T17:31:16Z

Pavel Zwerschke (@pavelzw) Sorry to ping you, is there anything we can do to help this get merged?

Pavel Zwerschke (pavelzw) · 2026-03-28T12:22:27Z

we now cleaned up and merged all other PRs and this one is next, sorry to keep you waiting for so long.

pixi lockfile v7 are coming soon-ish (prefix-dev/pixi#5607, conda/rattler#2026) and will lead to the version field not being fully required anymore.
we need to

merge main into this branch and resolve conflicts
see how the new lockfile format affects this PR.

Thibaut OPRINSEN (toprinse) · 2026-03-30T11:56:12Z

Hello Pavel Zwerschke (@pavelzw) .
So I merged the main branch into this one, fixed the conflicts and update tests.
I made a few changes :

I removed the win64 test (see comments in integrations_tests.rs because they were failing because the generated path from pixi build is too long. There is currently no clean solutions to avoid reaching these windows limitations.
Then, I'm not sure why but the test_pixi_pack_source sometimes fail, even I don't add major changes. I then get :

---- test_pixi_pack_source stdout ----
⏳ Downloading 25 packages...

thread 'test_pixi_pack_source' (13156) panicked at tests/integration_test.rs:858:5:
Err(could not download package: error decoding response body)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Maybe you have a better idea about what's happening ?

Pavel Zwerschke (pavelzw) · 2026-04-07T16:09:52Z

Then, I'm not sure why but the test_pixi_pack_source sometimes fail, even I don't add major changes

this looks like a networking issue and the package got dropped. maybe we need to add retries to the Client if we haven't done so already

Pavel Zwerschke (pavelzw) · 2026-05-04T06:23:24Z

Lockfile v7 is now on main (of pixi). This means we should be able to test against it :)

Pavel Zwerschke (pavelzw) · 2026-05-15T07:00:05Z

Added support for lockfile v7 in #306. hopefully this fixes our reproducibility issues

Thibaut OPRINSEN (toprinse) · 2026-05-27T09:23:56Z

Hello Pavel Zwerschke (@pavelzw). Sorry for the late answer. It's nice to have lockfiles v7 available !
Is there still things I can do for this PR ?

Pavel Zwerschke (pavelzw) · 2026-05-27T11:16:53Z

I think we would need to merge main and then check again whether the tests would now be working reliably

…ling

Thibaut OPRINSEN (toprinse) · 2026-05-27T14:43:41Z

I merged main into this branch and fixed the breaking changes introduced by the rattler_lock refactor (specifically, CondaSourceData.package_record was removed — replaced by record() returning an Option<&PackageRecord>). The tests appear to be passing now. Still I'm not sure they are working reliably. Seems that the merge did not affect packages names and hashes in the test.

Thibaut OPRINSEN (toprinse) requested review from Daniel Elsner (delsner) and Pavel Zwerschke (pavelzw) as code owners November 24, 2025 14:27

Thibaut OPRINSEN (toprinse) changed the title ~~feat : Add support for SOURCE packages~~ feat: Add support for SOURCE packages Nov 24, 2025

github-actions Bot added the enhancement New feature or request label Nov 24, 2025

feat: update pixi-pack to be able to pack packages from source

a532262

feat: try to add intergation tests

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from e214536 to faf6c1d Compare November 24, 2025 16:01

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from 6abfcac to a9b0110 Compare November 25, 2025 07:49

Daniel Elsner (delsner) requested changes Nov 28, 2025

View reviewed changes

Comment thread src/pack.rs Outdated

Comment thread src/pack.rs Outdated

Comment thread src/pack.rs Outdated

fix: remove recursive search and use packages from lock file

0847e0d

fix: fix typo fix: fix typo

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from abf424a to 0847e0d Compare November 29, 2025 13:22

Thibaut OPRINSEN (toprinse) requested a review from Daniel Elsner (delsner) November 29, 2025 13:22

Pavel Zwerschke (pavelzw) reviewed Nov 30, 2025

View reviewed changes

Comment thread src/pack.rs Outdated

fix: fix after review

1377324

fix: fix typo for ci precommit fix: fix typo for ci precommit fix: fix typo for ci precommit

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from a8e2dbc to 1377324 Compare December 1, 2025 12:04

Pavel Zwerschke (pavelzw) added 3 commits December 1, 2025 18:42

refactor local-build structure

482bd9e

explicitly drop

8d7d436

reword

4f94186

Pavel Zwerschke (pavelzw) reviewed Dec 1, 2025

View reviewed changes

fix: fix manifest_dir

177e276

fix: fix typo

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from 6a65aed to 177e276 Compare December 2, 2025 11:04

Pavel Zwerschke (pavelzw) mentioned this pull request Dec 18, 2025

pixi-build used dependencies (i.e. backend + host and build envs) should be persisted in pixi.lock prefix-dev/pixi#5161

Closed

Pavel Zwerschke (pavelzw) and others added 4 commits December 18, 2025 11:09

chore: Update shasums after release (Quantco#242)

456da43

ci: bump the gh-actions group with 5 updates (Quantco#247)

3506c08

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build: Bump rattler, use uv from crates.io (Quantco#243)

2f23e6e

fix: fix tests to match pixi 0.6.1

bea3f62

regenerate lock file to use pixi 0.61.0 fix: pin the build backend issue

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from 9301789 to bea3f62 Compare December 18, 2025 10:18

Thibaut OPRINSEN (toprinse) and others added 2 commits December 18, 2025 11:33

Merge branch 'main' into add_source_pkg_handling

b6b2895

fix: fix cargo lock

0384d5c

merge main into add_source_pkg_handling

e7eac88

feat: add source package handling with cross-platform fixes

eaf607c

Thibaut OPRINSEN (toprinse) force-pushed the add_source_pkg_handling branch from c17b233 to eaf607c Compare March 30, 2026 12:11

Thibaut Oprinsen added 2 commits May 27, 2026 15:26

Merge remote-tracking branch 'upstream/main' into add_source_pkg_hand…

5540214

…ling

fix: fix typo

f26f503

Conversation

Thibaut OPRINSEN (toprinse) commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 1, 2025

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Pavel Zwerschke (pavelzw) Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thibaut OPRINSEN (toprinse) Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 8, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 8, 2025

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 18, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 18, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Dec 18, 2025

Uh oh!

Thibaut OPRINSEN (toprinse) commented Dec 18, 2025

Uh oh!

vincent-pollet commented Feb 5, 2026

Uh oh!

vincent-pollet commented Mar 16, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented Mar 28, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented Mar 30, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented Apr 7, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 4, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 15, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented May 27, 2026

Uh oh!

Pavel Zwerschke (pavelzw) commented May 27, 2026

Uh oh!

Thibaut OPRINSEN (toprinse) commented May 27, 2026

Uh oh!

Reviewers

Assignees

Thibaut OPRINSEN (toprinse) commented Nov 24, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) Dec 1, 2025 •

edited

Loading

Thibaut OPRINSEN (toprinse) Dec 2, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) commented Dec 1, 2025 •

edited

Loading

Pavel Zwerschke (pavelzw) commented Dec 18, 2025 •

edited

Loading