Skip to content

Fix Bazel build errors for OSS environment#50

Open
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
czgdp1807:target-build
Open

Fix Bazel build errors for OSS environment#50
czgdp1807 wants to merge 1 commit intogoogle:masterfrom
czgdp1807:target-build

Conversation

@czgdp1807
Copy link
Contributor

Summary

This PR addresses multiple build failures when running bazel build //... in the open-source environment by removing Google-internal dependencies and fixing proto import paths.

Changes Made

Proto Library Support:

  • Added py_proto_library macro to struct2tensor.bzl for OSS compatibility with protobuf 4.x
  • Fixed proto import path in test_any.proto to use workspace-relative path (struct2tensor/test/any.proto)
  • Removed invalid load statements from non-existent third_party/protobuf/bazel and tools/build_defs/proto/cpp paths

Test Framework:

  • Replaced Google-internal gunit_main with OSS-compatible @com_google_googletest//:gtest_main in parquet tests
  • Commented out missing test files in kernels/parquet/BUILD:
    • parent_indices_builder_test.cc
    • parquet_reader_test.cc
    • parquet_reader_util_test.cc
  • Commented out serialization_benchmark that depends on internal benchmark framework

Dependencies:

  • Commented out Google-internal dependencies not available in OSS:
    • //testing/pybase and //testing/pybase:parameterized
    • //file/colossus/public:cns
    • //third_party/py/cpuinfo, //third_party/py/psutil, //third_party/py/yaml
    • //third_party/py/tensorflow_docs
    • //third_party/py/absl:app

Investigation Process

  1. Build error analysis: Bazel errors explicitly indicated missing packages:

    no such package 'testing/base/public'
    no such package 'third_party/py/psutil'
    no such package 'file/colossus/public'
    
  2. Workspace verification: Checked third_party/ directory structure and confirmed these paths don't exist in the OSS repository

  3. Dependency pattern recognition: Distinguished internal vs OSS dependencies:

    • Internal: //path/to/internal (workspace-relative paths that don't exist)
    • OSS: @external_repo//path (external dependencies like @absl_py//, @com_google_protobuf//)
  4. Iterative testing: Confirmed by progressively commenting out suspected internal dependencies and verifying build progress

Testing

  • Verified bazel build //... completes successfully in OSS build environment
  • All non-commented targets build without errors

Impact

  • Enables successful Bazel builds for OSS contributors
  • No functional changes to existing code paths
  • Internal dependencies are clearly marked with comments for future reference

- Add py_proto_library macro for OSS compatibility
- Replace gunit_main with googletest gtest_main
- Comment out Google-internal dependencies and missing test files
- Fix proto import to use workspace-relative path
- Remove invalid third_party load statements

Fixes build errors when running `bazel build //...` in OSS environment.
@czgdp1807
Copy link
Contributor Author

Here are the build logs,

(s2t-env) czgdp1807@qgpu3:~/struct2tensor$ bazel build /...
INFO: Build option --action_env has changed, discarding analysis cache.
WARNING: /home/czgdp1807/.cache/bazel/_bazel_czgdp1807/38445803433c11c40285b6c944f811b3/external/local_config_tf/BUILD:20551:8: target 'libtensorflow_framework.so' is both a rule and a file; please choose another name for the rule
INFO: Analyzed 173 targets (199 packages loaded, 12848 targets configured).
INFO: Found 173 targets...
INFO: From Compiling snappy-sinksource.cc:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling snappy-stubs-internal.cc:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling snappy.cc:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling src/google/protobuf/compiler/rust/relative_path.cc [for tool]:
external/com_google_protobuf/src/google/protobuf/compiler/rust/relative_path.cc: In member function 'std::string google::protobuf::compiler::rust::RelativePath::Relative(const google::protobuf::compiler::rust::RelativePath&) const':
external/com_google_protobuf/src/google/protobuf/compiler/rust/relative_path.cc:66:21: warning: comparison of integer expressions of different signedness: 'int' and 'std::vector<std::basic_string_view<char> >::size_type' {aka 'long unsigned int'} [-Wsign-compare]
   66 |   for (int i = 0; i < current_segments.size(); ++i) {
      |                   ~~^~~~~~~~~~~~~~~~~~~~~~~~~
INFO: From Compiling src/google/protobuf/arena.cc [for tool]:
external/com_google_protobuf/src/google/protobuf/arena.cc: In member function 'void* google::protobuf::internal::SerialArena::AllocateAlignedFallback(size_t)':
external/com_google_protobuf/src/google/protobuf/arena.cc:200:10: warning: 'ret' may be used uninitialized [-Wmaybe-uninitialized]
  200 |   return ret;
      |          ^~~
external/com_google_protobuf/src/google/protobuf/arena.cc:197:9: note: 'ret' was declared here
  197 |   void* ret;
      |         ^~~
INFO: From Compiling snappy-sinksource.cc [for tool]:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling src/google/protobuf/message_lite.cc [for tool]:
external/com_google_protobuf/src/google/protobuf/message_lite.cc: In member function 'int google::protobuf::MessageLite::GetCachedSize() const':
external/com_google_protobuf/src/google/protobuf/message_lite.cc:69:70: warning: 'int google::protobuf::MessageLite::ByteSize() const' is deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations]
   69 |   if (PROTOBUF_PREDICT_FALSE(cached_size == nullptr)) return ByteSize();
      |                                                              ~~~~~~~~^~
In file included from external/com_google_protobuf/src/google/protobuf/message_lite.cc:13:
bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/message_lite.h:454:59: note: declared here
  454 |   [[deprecated("Please use ByteSizeLong() instead")]] int ByteSize() const {
      |                                                           ^~~~~~~~
INFO: From Compiling snappy-stubs-internal.cc [for tool]:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling snappy.cc [for tool]:
cc1plus: warning: command-line option '-Wno-implicit-function-declaration' is valid for C/ObjC but not for C++
INFO: From Compiling src/google/protobuf/generated_message_tctable_lite.cc [for tool]:
In file included from bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/generated_message_tctable_decl.h:22,
                 from external/com_google_protobuf/src/google/protobuf/generated_message_tctable_lite.cc:22:
bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/parse_context.h:1125:1: warning: 'always_inline' function might not be inlinable [-Wattributes]
 1125 | ParseContext::ParseLengthDelimitedInlined(const char* ptr, const Func& func) {
      | ^~~~~~~~~~~~
bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite/google/protobuf/parse_context.h:1125:1: warning: 'always_inline' function might not be inlinable [-Wattributes]
external/com_google_protobuf/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: 'always_inline' function might not be inlinable [-Wattributes]
  830 | PROTOBUF_ALWAYS_INLINE const char* TcParser::FastVarintS1(
      |                                    ^~~~~~~~
external/com_google_protobuf/src/google/protobuf/generated_message_tctable_lite.cc:830:36: warning: 'always_inline' function might not be inlinable [-Wattributes]
external/com_google_protobuf/src/google/protobuf/generated_message_tctable_lite.cc:753:29: warning: 'always_inline' function might not be inlinable [-Wattributes]
  753 | PROTOBUF_ALWAYS_INLINE bool EnumIsValidAux(int32_t val, uint16_t xform_val,
      |                             ^~~~~~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/generated_message_tctable_lite.cc:741:29: warning: 'always_inline' function might not be inlinable [-Wattributes]
  741 | PROTOBUF_ALWAYS_INLINE void PrefetchEnumData(uint16_t xform_val,
      |                             ^~~~~~~~~~~~~~~~
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/any_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/descriptor_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/compiler/plugin_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/duration_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/empty_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/api_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/field_mask_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/struct_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/source_context_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/type_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/timestamp_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From ProtoCompile external/com_google_protobuf/python/google/protobuf/wrappers_pb2.py:
external/com_google_protobuf/.: warning: directory does not exist.
INFO: From Generating Descriptor Set proto_library @com_github_tensorflow_metadata//tensorflow_metadata/proto/v0:metadata_v0_proto:
tensorflow_metadata/proto/v0/metric.proto:23:1: warning: Import tensorflow_metadata/proto/v0/path.proto is unused.
tensorflow_metadata/proto/v0/problem_statement.proto:20:1: warning: Import google/protobuf/struct.proto is unused.
INFO: From Generating proto_library @com_github_tensorflow_metadata//tensorflow_metadata/proto/v0:metadata_v0_proto:
tensorflow_metadata/proto/v0/metric.proto:23:1: warning: Import tensorflow_metadata/proto/v0/path.proto is unused.
tensorflow_metadata/proto/v0/problem_statement.proto:20:1: warning: Import google/protobuf/struct.proto is unused.
INFO: From Compiling tsl/platform/status.cc [for tool]:
external/local_tsl/tsl/platform/status.cc: In function 'std::vector<tsl::StackFrame> tsl::errors::GetStackTrace(const absl::lts_20230802::Status&)':
external/local_tsl/tsl/platform/status.cc:146:23: warning: comparison of integer expressions of different signedness: 'int' and 'std::vector<std::__cxx11::basic_string<char> >::size_type' {aka 'long unsigned int'} [-Wsign-compare]
  146 |     for (int i = 0; i < split.size() / 3; ++i) {
      |                     ~~^~~~~~~~~~~~~~~~~~
INFO: Elapsed time: 154.236s, Critical Path: 146.36s
INFO: 2151 processes: 503 internal, 1648 processwrapper-sandbox.
INFO: Build completed successfully, 2151 total actions

@czgdp1807
Copy link
Contributor Author

@vkarampudi If you want anything to be uncommented please let me know. I will see what I can do. There are a few things which I concluded to be Google-internal. My reasoning may be wrong, so please let me know in that case.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant