Explicitly build dynamic op libraries for wheel packaging#51
Open
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
Open
Explicitly build dynamic op libraries for wheel packaging#51czgdp1807 wants to merge 1 commit intogoogle:masterfrom
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
Conversation
Add explicit build and copy steps for dynamic op libraries (.so files) in
build_common.sh. This ensures all six dynamic library targets are built and
copied to struct2tensor/ops/ before the wheel is packaged.
Previously, the stamp_wheel step in build_manylinux.sh would fail with
patchelf errors ("No such file or directory") because the .so files were
expected in struct2tensor/ops/ but were not being copied from bazel-bin.
This fix:
- Explicitly builds each dynamic library target
- Copies the built .so files from bazel-bin to the staging directory
- Runs before bazel run :build_pip_package to ensure files are present
Fixes the missing _decode_proto_map_op.so, _decode_proto_sparse_op.so and
related dynamic library warnings during wheel packaging.
Contributor
Author
|
Here are the build logs, .
.
.
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_decode_proto_map_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_decode_proto_map_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_decode_proto_map_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_decode_proto_sparse_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_decode_proto_sparse_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_decode_proto_sparse_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_run_length_before_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_run_length_before_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_run_length_before_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_equi_join_any_indices_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_equi_join_any_indices_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_equi_join_any_indices_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_equi_join_indices_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_equi_join_indices_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_equi_join_indices_op.so
+ for SO_FILE in "${libraries[@]}"
+ SO_FILE_PATH=struct2tensor/ops/_parquet_dataset_op.so
+ cp /tmp/tmp.qOjY11CtU8/struct2tensor/ops/_parquet_dataset_op.so /tmp/tmp.hWCK9OOEW6/struct2tensor/ops/_parquet_dataset_op.so
+ rm /struct2tensor/dist/struct2tensor-0.49.0.dev0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ wheel version
wheel 0.46.3
+ wheel pack /tmp/tmp.hWCK9OOEW6 --dest-dir /struct2tensor/dist
Repacking wheel as /struct2tensor/dist/struct2tensor-0.49.0.dev0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl...OK |
Contributor
Author
|
After merging this PR and #50 in my |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When building the struct2tensor wheel,
patchelfwarnings occur during thestamp_wheelstep,The
stamp_wheelfunction expects all six dynamic op library files (.so) to be present instruct2tensor/ops/before attempting to process them, but they were not being built and copied from Bazel's output directory.Root Cause
With the upgrade to Protobuf 4.25.6, the build system changed:
protobuf.bzlprovided public macros (cc_proto_library,py_proto_library,proto_genrule withgen_ccattribute) that automatically compiled.protofiles to C++.pb.cc/.pb.hfiles. These were used by dynamic library targets._proto_gen,internal_py_proto_library). The oldproto_genrule also removed thegen_ccattribute, making C++ generation implicit and harder to trigger.Bottom line: Protobuf 4.25.6 removed the public build rule APIs that struct2tensor's proto dependencies relied on. Without access to
cc_proto_libraryfromprotobuf.bzl, the proto files in struct2tensor couldn't be compiled to C++, preventing the dynamic libraries from being built.Solution
Add an explicit
build_dynamic_libraries()function inbuild_common.shthat:bazel build.sofiles frombazel-bin/struct2tensor/ops/tostruct2tensor/ops/bazel run :build_pip_packagestep to ensure files are staged for wheel creationThis ensures the
.sofiles are present and valid beforestamp_wheelattempts to patch them.Files Changed
build_dynamic_libraries()function and integrated it into the build flowTesting
Wheel builds successfully with all dynamic libraries properly staged and patched.