fix(macos): make matching python3-config available in build/test venvs#2922
Draft
henryiii wants to merge 3 commits into
Draft
fix(macos): make matching python3-config available in build/test venvs#2922henryiii wants to merge 3 commits into
henryiii wants to merge 3 commits into
Conversation
Virtualenvs do not contain a python3-config (or versioned
python{X.Y}-config) script, and cibuildwheel only puts the venv bin
directory on PATH. The python.org framework builds ship these scripts
next to the base interpreter, but that directory is never added to PATH,
so C/C++ extensions that invoke python3-config picked up an unrelated
(system/Homebrew) python3-config, producing broken or ABI-mismatched
wheels.
After creating the venv on POSIX platforms, symlink the base
interpreter's python*-config scripts into the venv bin so the correct
one is found first on PATH. Interpreters without such a script (PyPy,
GraalPy, some uv-managed builds) are handled gracefully and never fail
the build.
Assisted-by: ClaudeCode:claude-opus-4.8
Fixes ruff (ANN/TC) and mypy (no-untyped-def) failures in CI. Assisted-by: ClaudeCode:claude-opus-4.8
4b8d3a5 to
968ce48
Compare
Refactor _symlink_python_config_scripts docstring to clarify behavior regarding linking python-config scripts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Problem
On macOS, cibuildwheel builds and tests inside a virtualenv and only adds the venv's
bindirectory toPATH. Virtualenvs do not contain apython3-config(or versionedpython{X.Y}-config) script. The python.org framework installs (/Library/Frameworks/Python.framework/Versions/{version}/bin/) do shippython3-config, but that directory is never placed onPATH.As a result, complex C/C++ extension builds that invoke
python3-config(to obtain include dirs / link flags) fall through to an unrelatedpython3-configfound elsewhere onPATH— typically the system or Homebrew Python (e.g./opt/homebrew/bin/python3-config). This produces broken wheels or ABI mismatches.Root cause
cibuildwheel/venv.pycreates the venv andactivate_virtualenvonly prepends the venvbintoPATH. Nopython*-configexists in thatbin, and the matching one next to the base interpreter is not exposed.Fix
After the venv is created (POSIX only), symlink the base interpreter's
python*-configscripts into the venv'sbindirectory so the correct one is found first onPATH. This is done in a small helper_symlink_python_config_scripts(base_python, venv_bin)called fromvirtualenv().python*-configglob next to the base interpreter, so both the unversionedpython3-configand the versionedpython3.X-configare linked when present.binare never overwritten.OSErrorduring symlink creation is suppressed, so the build is never failed by this best-effort step.Edge cases / limitations
configscripts are a POSIX concept; guarded by_IS_WIN).pypy3-config/ similar, which thepython*-configglob does not match, so nothing is linked — by design. The build is never failed when no config script exists.virtualenv()callers (macOS, iOS, Android, pyodide, audit), but only acts when a matchingpython*-configactually exists next to the base interpreter, so non-macOS flows are effectively unaffected.Testing
Added
unit_test/venv_test.pycovering the helper directly (the fullvirtualenv()path requires network/installs and a real framework Python, so a true end-to-end unit test is impractical):python3-config+python3.X-configgets both linked into the venvbin;pypy3-config) links nothing;python3-configin the venvbinis left untouched.All three pass; the full
unit_testsuite passes (791 passed).ruff,ruff-format,codespell, andflake8-lazypass on the changed files; mypy is clean onvenv.pyapart from the pre-existinguvimport-stub note that the project config already ignores.Note for reviewers: the helper is best-effort (uses symlinks, suppresses
OSError). Open question: should it prefer copying over symlinking, or restrict the behavior to the CPython/macOS path only rather than all POSIXvirtualenv()callers? Happy to narrow the scope if preferred.Closes #2021