Skip to content

HIVE-29262: [Addendum] Fix VectorPTFGroupBatches partition key copy when partition and order columns overlap#6597

Open
Aggarwal-Raghav wants to merge 1 commit into
apache:masterfrom
Aggarwal-Raghav:ptf_overlap
Open

HIVE-29262: [Addendum] Fix VectorPTFGroupBatches partition key copy when partition and order columns overlap#6597
Aggarwal-Raghav wants to merge 1 commit into
apache:masterfrom
Aggarwal-Raghav:ptf_overlap

Conversation

@Aggarwal-Raghav

@Aggarwal-Raghav Aggarwal-Raghav commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This addendum provides a better fix in my opinion for the original issue and another issue i.e. ClassCastException in VectorPTFGroupBatches when partition and order columns overlap, replacing the original approach taken in #6512
Check the discussion here: #6512 (comment)

Why are the changes needed?

This PR, is also fixing 1 additional issue of ClassCastException when there is overlap column in partition by and order by clause. I observed the below stacktrace on Hive 4.0.1 after debugging I found this PR is fixing it as well.

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 8)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:410)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:258)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
    ... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 8)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:518)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)
    ... 19 more
Caused by: java.lang.ClassCastException: class [B cannot be cast to class java.lang.Long ([B and java.lang.Long are in module java.base of loader 'bootstrap')
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.setRepeatingColumn(VectorPTFGroupBatches.java:803)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.copyPartitionColumnToOverflow(VectorPTFGroupBatches.java:785)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.forwardSpilledBatches(VectorPTFGroupBatches.java:723)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.fillGroupResultsAndForward(VectorPTFGroupBatches.java:525)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.finishPartition(VectorPTFOperator.java:633)
    at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:402)
    at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
    at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:502)
    ... 20 more

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Wrote a new q file and original q file introduced in older approach is also passing.

mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver -Dqfile=vector_ptf_cols_order.q -Dtest.output.overwrite=true -Drat.skip=true
mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver -Dqfile=vector_ptf_spill_partition_order_overlap.q -Dtest.output.overwrite=true -Drat.skip=true

@sonarqubecloud

Copy link
Copy Markdown

@Aggarwal-Raghav Aggarwal-Raghav marked this pull request as ready for review July 10, 2026 15:10

@deniskuzZ deniskuzZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants