Ensure python_wheel sources are linked after dependent targets are built

[marcuscaisey]

Problem

python_wheel sources are not symlinked into plz-out/python/venv after dependent targets are built.

Worked example

After running the following commands in this repo:

plz clean --nobackground
plz test //test:name_scheme_test

When I open the test file test/name_scheme_test.py:

My language server can't resolve the sources from either of the python_wheel targets //third_party/python:click and //third_party/python:click-log that //test:name_scheme_test depends on. I've configured it to look in plz-out/python/venv for modules, but the sources aren't there.

Expected Behaviour

After building a python_xxx target which depends on a python_wheel target, the sources of the python_wheel have been symlinked into plz-out/python/venv.

Actual Behaviour

The sources are not symlinked unless //test:name_scheme_test is built directly. This is annoying and usually results in me just doing plz build //third_party/python/... to link everything even though not all of the targets are required. This behaviour doesn't align with proto_library, go_repo, or pip_library.

Cause

While //third_party/python:click has the appropriate link:plz-out/python/venv label:

$ plz query print //third_party/python:click
# //third_party/python:click:
build_rule(
    ...
    labels = [
        ...
        'link:plz-out/python/venv',
    ],
    ...
    provides = {
        'py': ['//third_party/python:_click#wheel'],
        ...
    },
    ...
)

//test:name_scheme_test depends on //third_party/python:_click#wheel instead because //test:_name_scheme_test#pex has a py dependency requirement:

$ plz query print //test:_name_scheme_test#pex
# //test:_name_scheme_test#pex:
build_rule(
    ...
    deps = [
        '//third_party/python:click',
        ...
    ],
    ...
    requires = ['py'],
)

$ plz query deps //test:name_scheme_test --hidden
...
//test:_name_scheme_test#lib
  ...
  //test:_name_scheme_test#pex
    //third_party/python:_click#wheel
      ...
    ...

So //test:name_scheme_test never gets built.

Solution

Split //third_party/python:_click#wheel into //third_party/python:_click#srcs and //third_party/python:_click#wheel:

• :_click#srcs does everything that :_click#wheel did except create the .pex.zip. This target is labelled with link:plz-out/python/venv instead of :click.
• //test:_name_scheme_test#wheel now just zips up the output of //test:_name_scheme_test#srcs. The output of this target is unchanged.
• :click is now a filegroup which exposes _click#srcs. The output of this target is unchanged, however it now doesn't depend on _click#wheel as it did before (see bonus change for more on this).

Now, regardless of whether a target depends on //third_party/python:_click#wheel or //third_party/python:click, the sources will be symlinked into plz-out:

$ plz clean --nobackground
$ plz test //test:name_scheme_test
//test:name_scheme_test 2 tests run in 1.086s; 2 passed
    test_click_is_importable      PASS  6ms
    test_click_log_is_importable  PASS  8ms
1 test target and 2 tests run; 2 passed.
Total time: 4.34s real, 1.09s compute.

$ ls plz-out/python/venv
click  click_log  portalocker  xmlrunner

Bonus Change

After updating python_wheel, the following test failed to build:

$ plz test //test:namespaced_packages_test
Build stopped after 4.77s. 1 target failed:
    //test:_namespaced_packages_test#pex
cannot calculate hash for plz-out/gen/third_party/python/setuptools.pex.zip: file does not exist

The root cause is thought-machine/please#3531.

plz-out/gen/third_party/python/setuptools.pex.zip is the output of //third_party/python:_setuptools#wheel which was previously being built because //test:_namespaced_packages_test#pex depends on //third_party/python:_setuptools#wheel transitively through //test:_namespaced_packages_test#pex -> //third_party/python:googleapis_common_protos -> //third_party/python:protobuf_pip -> //third_party/python:setuptools -> //third_party/python:_setuptools#wheel.

After the python_wheel change, //third_party/python:setuptools doesn't depend on //third_party/python:_setuptools#wheel so its output is not available when //test:_namespaced_packages_test#pex is being built.

To get around this, i've added the py requirement to the main pip_library target (which //third_party/python:protobuf_pip is) so that it depends on //third_party/python:_setuptools#wheel instead of //third_party/python:setuptools. I've also excluded .pex.zip files from the .whl output by //third_party/python:protobuf_pip. Overall, this is an improvement because //third_party/python:protobuf_pip was including all of the setuptools sources in its .whl.

Before:

$ unzip -l $(plz build //third_party/python:protobuf_pip) | rg setuptools --count
621

After:

$ unzip -l $(plz build //third_party/python:protobuf_pip) | rg setuptools --count

I've included this change as a separate commit.

[marcuscaisey]

The FreeBSD tests will fail because the ports need to be updated. I did that in #297

[marcuscaisey]

This was never needed since the build doesn't make any network requests

[marcuscaisey]

I think this logic was trying to handle the scenario where the sources are output into the same directory as one of the deps. The #srcs rule doesn't define any deps, so this isn't required anymore.