[spark] Make DeletionVectorTest pass with write.use-v2-write enabled#8614
Merged
Zouxxyy merged 1 commit intoJul 14, 2026
Merged
Conversation
2ddf56a to
174f9d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
DeletionVectorTestfails when the suite is run withspark.paimon.write.use-v2-write=true.Its helper
runAndCheckSplitScanasserts that a DELETE/UPDATE/MERGE on a deletion-vector append table reads through aPaimonSplitScan(the V1 deletion-vector copy-on-write path). However, since #8539 added V2 delta-based row-level operations, an unaware-bucket deletion-vector append table with V2 write enabled is routed toPaimonSparkDeltaOperation/WriteDelta, whose scan side is a regular pushdown scan (seePaimonSparkDeltaOperation.newScanBuilder) and therefore produces noPaimonSplitScan. Theassert(batchScans.nonEmpty)then fails:With the default (
write.use-v2-write=false) the suite still exercises the V1 path, so the failure is only visible when V2 write is enabled. The delta path itself is functionally correct and covered byDeltaRowLevelOpsTest.Tests
runAndCheckSplitScannow detects the V2 delta plan (WriteDelta, viaEXPLAIN EXTENDED) and, in that case, just executes the statement instead of asserting the V1 split scan. Behavior is unchanged for the V1 path (default config and bucketed deletion-vector append tables), and the suite now passes under bothwrite.use-v2-write=falseandtrue.API and Format
No public API or format change.
Documentation
No documentation change.