[MINOR] Decouple BHJ/BNLJ type flag from broadcast build-side cache key by philo-he · Pull Request #12309 · apache/gluten

philo-he · 2026-06-16T23:38:58Z

What changes are proposed in this pull request?

Rename BroadCastHashJoinContext → BroadcastJoinContext and BroadCastJoinBuilder → BroadcastJoinBuilder to fix the camelCase typos consistently across Scala, Java, and C++ files.
Replace string-prefix join type detection with an explicit isBhj flag — StorageJoinBuilder previously distinguished BHJ from BNLJ by checking buildHashTableId.startsWith("BuiltBNLJBroadcastTable-"), which was fragile. The new isBhj: Boolean field in BroadcastJoinContext makes the intent explicit.

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

Relied on Claude fixing ClickHouse backend failures.

github-actions · 2026-06-16T23:39:37Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

This PR refactors broadcast-join plumbing across Gluten’s Substrait layer and ClickHouse backend by switching broadcast table/hash-table identifiers from string keys to numeric IDs, and by unifying “BroadCast*” naming to “Broadcast*”. It also updates the ClickHouse JNI + native join builder path to carry an explicit “is BHJ” flag instead of inferring join type from string prefixes.

Changes:

Change join parameter generation to emit an integer buildHashTableId / buildTableId (instead of prefixed strings).
Rename ClickHouse broadcast join context/builder symbols from BroadCast* to Broadcast*, and update JNI/native builder APIs accordingly.
Adjust benchmark plan/test code to match the new identifier format.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
gluten-substrait/src/main/scala/org/apache/gluten/execution/JoinExecTransformer.scala	Updates join-parameter internal API to use an `Int` build table ID.
gluten-substrait/src/main/scala/org/apache/gluten/execution/BroadcastNestedLoopJoinExecTransformer.scala	Switches broadcast nested-loop build table ID to `Int`.
backends-velox/src/main/scala/org/apache/gluten/execution/HashJoinExecTransformer.scala	Implements new join-parameter internal signature for Velox SHJ.
backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHHashJoinExecTransformer.scala	Renames/reshapes broadcast join context type for ClickHouse.
backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHBroadcastNestedLoopJoinExecTransformer.scala	Updates CH BNLJ context + join-parameter emission for native side.
backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHBroadcastBuildSideCache.scala	Changes cache key type from `String` to `Int`.
backends-clickhouse/src/main/scala/org/apache/spark/sql/execution/joins/ClickHouseBuildSideRelation.scala	Migrates build-side relation to use the new broadcast context type.
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/StorageJoinBuilder.java	Updates JNI signatures and passes explicit `isBhj` to native builder.
cpp-ch/local-engine/local_engine_jni.cpp	Updates native JNI entrypoints for the new Java signatures and builder name.
cpp-ch/local-engine/Join/BroadcastJoinBuilder.h	Renames namespace and updates APIs to use numeric keys + explicit `is_bhj`.
cpp-ch/local-engine/Join/BroadcastJoinBuilder.cpp	Implements new API: uses `is_bhj` instead of string prefix checks.
cpp-ch/local-engine/Parser/AdvancedParametersParseUtil.h	Changes `storage_join_key` to an integer type.
cpp-ch/local-engine/Parser/RelParsers/JoinRelParser.cpp	Renames builder usage to `BroadcastJoinBuilder`.
cpp-ch/local-engine/Parser/RelParsers/CrossRelParser.cpp	Renames builder usage to `BroadcastJoinBuilder`.
cpp/velox/benchmarks/data/plan/q17_joins.json	Updates plan data to use numeric `buildHashTableId`.
backends-clickhouse/src/test/scala/org/apache/spark/sql/execution/benchmarks/CHStorageJoinBenchmark.scala	Renames local variables to `broadcast*` spelling.
backends-clickhouse/src/test/scala/org/apache/spark/sql/execution/benchmarks/CHHashBuildBenchmark.scala	Updates imports/types and JNI call to numeric hash-table ID.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+  // isBHJ, isNullAwareAntiJoin, buildTableId.
+  def genJoinParametersInternal(): (Int, Int, Int)


  private var existHashTableData: Long = 0L
  private var existBroadCastHashJoinContext: BroadCastHashJoinContext = null


  }

-  private def couldReuseHashTableData(broadCastContext: BroadCastHashJoinContext): Boolean = {
+  private def couldReuseHashTableData(broadcastContext: BroadCastHashJoinContext): Boolean = {


    buildSideRelationCache
      .get(
-        broadCastContext.buildHashTableId,
-        (broadcast_id: String) => {
+        broadcastContext.buildTableId,
+        (broadcastId: String) => {
          val (pointer, relation) =


    joinParametersStr
      .append("isBHJ=")
-      .append(1)
+      .append(0)
      .append("\n")


+      .append(0)
      .append("\n")
-      .append("buildHashTableId=")
+      .append("buildBroadcastTableId=")


    Int64 right_table_bytes = -1;
    Int64 partitions_num = -1;
-    String storage_join_key;
+    Int32 storage_join_key = -1;



github-actions · 2026-06-17T00:22:03Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-17T03:52:23Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 6 comments.

github-actions · 2026-06-17T05:43:04Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-17T22:57:34Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-17T23:23:18Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-17T23:50:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-18T04:38:15Z

Run Gluten Clickhouse CI on x86

philo-he · 2026-06-18T04:53:39Z

@zzcclp, could you take a look?

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

      .get(
-        broadCastContext.buildHashTableId,
-        (broadcast_id: String) => {
+        broadcastContext.buildTableId,
+        (broadcastId: String) => {
          val (pointer, relation) =
            broadcast.value
              .asInstanceOf[ClickHouseBuildSideRelation]
-              .buildHashTable(broadCastContext)
-          logDebug(s"Create bhj $broadcast_id = 0x${pointer.toHexString}")
+              .buildHashTable(broadcastContext)
+          logDebug(s"Create bhj $broadcastId = 0x${pointer.toHexString}")
          BroadcastHashTable(pointer, relation)


  override def genJoinParameters(): Any = {
-    val (isBHJ, isNullAwareAntiJoin, buildHashTableId): (Int, Int, String) = (0, 0, "")
+    val (isBHJ, isNullAwareAntiJoin, buildHashTableId): (Int, Int, Int) = (0, 0, 0)



    joinParametersStr
      .append("isBHJ=")
-      .append(1)
+      .append(0)
      .append("\n")
-      .append("buildHashTableId=")
+      .append("buildBroadcastTableId=")
      .append(buildBroadcastTableId)
      .append("\n")


 JNIEXPORT jlong JNICALL Java_org_apache_gluten_vectorized_HashJoinBuilder_nativeBuild( // NOLINT
    JNIEnv* env,
    jobject wrapper,
-    jstring tableId,
+    jint tableId,
    jlongArray batchHandles,
    jobjectArray joinKeys,
    jobjectArray filterBuildColumns,


github-actions · 2026-06-18T05:50:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-18T17:09:30Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

  override def genJoinParameters(): Any = {
-    val (isBHJ, isNullAwareAntiJoin, buildHashTableId): (Int, Int, String) = (0, 0, "")
+    val (isBHJ, isNullAwareAntiJoin, buildHashTableId): (Int, Int, Int) = (0, 0, 0)


    JNIEnv* env,
    jobject wrapper,
-    jstring tableId,
+    jint tableId,


+  private def couldReuseHashTableData(broadcastContext: BroadcastJoinContext): Boolean = {
    if (existHashTableData != 0) {
-      existBroadCastHashJoinContext.joinType == broadCastContext.joinType &&
-      existBroadCastHashJoinContext.hasMixedFiltCondition == broadCastContext.hasMixedFiltCondition
+      existBroadcastHashJoinContext.joinType == broadcastContext.joinType &&
+      existBroadcastHashJoinContext.hasMixedFiltCondition == broadcastContext.hasMixedFiltCondition
    }


github-actions · 2026-06-18T21:56:39Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-18T22:08:49Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-18T22:11:45Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

      .append("isBHJ=")
-      .append(1)
+      .append(0)


JkSelf · 2026-06-19T02:16:20Z

@philo-he Thanks for your work.

We are working on using Velox’s HashTableCache to cache pre-built hash tables here so we can revert a custom commit in our internal Velox repo. Since HashTableCache currently uses String as the cache key, can we continue using a string for our cache key.

github-actions · 2026-06-19T04:28:31Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

philo-he · 2026-06-19T04:59:29Z

@philo-he Thanks for your work.

We are working on using Velox’s HashTableCache to cache pre-built hash tables here so we can revert a custom commit in our internal Velox repo. Since HashTableCache currently uses String as the cache key, can we continue using a string for our cache key.

@JkSelf, thanks for the comment.
The String key was previously used both as a cache key and as a type descriptor (via starts_with("BuiltBNLJBroadcastTable-")). This PR separates them — the isBhj flag is used to distinguish join operator type, so the key only needs to be a unique identifier.

In theory, we can keep using string key in this PR, converted from build plan ID. Let me think about this. And even if Gluten finally uses Int key, we still can convert it to string at the Velox boundary. So I think you can keep using the string cache key in that Velox PR.

github-actions · 2026-06-22T06:26:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-22T06:33:57Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

      logWarning(
-        s"Can't not trace broadcast table data $buildBroadcastTableId" +
+        s"Can not trace broadcast table data $buildBroadcastTableId" +
          s" because execution id is null." +


  private static native long nativeBuild(
-      String buildHashTableId,
+      String buildTableId,
      byte[] in,
      long rowCount,
      String joinKeys,


github-actions · 2026-06-22T16:52:23Z

Run Gluten Clickhouse CI on x86

philo-he · 2026-06-22T20:43:40Z

@JkSelf, I have reverted the changes for broadcast ID type – keep using String type in this PR. Could you take a look gain?

Also cc @zzcclp since there are a few code changes related to ClickHouse backend.

JkSelf

LGTM. Thanks for your refactor.

Copilot AI review requested due to automatic review settings June 16, 2026 23:38

github-actions Bot added CORE works for Gluten Core VELOX CLICKHOUSE labels Jun 16, 2026

Copilot started reviewing on behalf of philo-he June 16, 2026 23:39 View session

philo-he mentioned this pull request Jun 16, 2026

[CORE] Refactor broadcast join related code #9439

Closed

Copilot AI reviewed Jun 16, 2026

View reviewed changes

philo-he marked this pull request as draft June 17, 2026 00:10

philo-he requested a review from Copilot June 17, 2026 03:59

Copilot started reviewing on behalf of philo-he June 17, 2026 04:00 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

philo-he force-pushed the refactor-broadcast-join branch from facb2d1 to 58d4c0f Compare June 17, 2026 23:22

philo-he marked this pull request as ready for review June 18, 2026 04:53

Copilot AI review requested due to automatic review settings June 18, 2026 04:53

Copilot started reviewing on behalf of philo-he June 18, 2026 04:53 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 18, 2026 17:08

Copilot started reviewing on behalf of philo-he June 18, 2026 17:09 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

philo-he force-pushed the refactor-broadcast-join branch from 5931c1d to aa3af94 Compare June 18, 2026 21:56

Copilot AI review requested due to automatic review settings June 18, 2026 22:08

philo-he force-pushed the refactor-broadcast-join branch from aa3af94 to 34d6723 Compare June 18, 2026 22:08

Copilot started reviewing on behalf of philo-he June 18, 2026 22:08 View session

philo-he force-pushed the refactor-broadcast-join branch from 34d6723 to 9d29f03 Compare June 18, 2026 22:11

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread ...se/src/main/scala/org/apache/gluten/execution/CHBroadcastNestedLoopJoinExecTransformer.scala

Comment on lines 101 to +102

.append("isBHJ=")

.append(1)

.append(0)

Copilot AI review requested due to automatic review settings June 19, 2026 04:28

Copilot started reviewing on behalf of philo-he June 19, 2026 04:28 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

philo-he changed the title ~~[CORE] Refactor broadcast join context and table ID type~~ [MINOR] Refactor broadcast join context and table ID type Jun 19, 2026

philo-he changed the title ~~[MINOR] Refactor broadcast join context and table ID type~~ [MINOR] Decouple BHJ/BNLJ type flag from broadcast build-side cache key Jun 22, 2026

philo-he force-pushed the refactor-broadcast-join branch from 8489dbd to 17d0b75 Compare June 22, 2026 06:33

Copilot AI review requested due to automatic review settings June 22, 2026 06:33

Copilot started reviewing on behalf of philo-he June 22, 2026 06:34 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Initial

c96a4d6

philo-he force-pushed the refactor-broadcast-join branch from 17d0b75 to c96a4d6 Compare June 22, 2026 16:51

JkSelf approved these changes Jun 23, 2026

View reviewed changes

philo-he merged commit 1d09d32 into apache:main Jun 23, 2026
71 checks passed

philo-he deleted the refactor-broadcast-join branch June 23, 2026 16:52

		// isBHJ, isNullAwareAntiJoin, buildTableId.
		def genJoinParametersInternal(): (Int, Int, Int)

		private var existHashTableData: Long = 0L
		private var existBroadCastHashJoinContext: BroadCastHashJoinContext = null

Uh oh!

Conversation

philo-he commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

philo-he commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

JkSelf commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

philo-he commented Jun 19, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

Copilot AI left a comment

philo-he commented Jun 16, 2026 •

edited

Loading

JkSelf commented Jun 19, 2026 •

edited

Loading

philo-he commented Jun 22, 2026 •

edited

Loading