feat(isthmus)!: remove invalid SetOp to Calcite relation mappings#998
Open
nielspardon wants to merge 1 commit into
Open
feat(isthmus)!: remove invalid SetOp to Calcite relation mappings#998nielspardon wants to merge 1 commit into
nielspardon wants to merge 1 commit into
Conversation
The Substrait -> Calcite converter mapped MINUS_MULTISET to Calcite Minus(all=true) and INTERSECTION_PRIMARY to Intersect(all=false). These mappings are incorrect: the spec behaviour of these set operations does not match the Calcite relations they were mapped to. They were retained temporarily to let users migrate off of them. Neither MINUS_MULTISET nor INTERSECTION_PRIMARY has an equivalent Calcite relation, so converting a plan containing either to Calcite now throws UnsupportedOperationException. BREAKING CHANGE: Converting a Substrait plan that uses the MINUS_MULTISET or INTERSECTION_PRIMARY set operations to Calcite now throws UnsupportedOperationException instead of producing an incorrect Calcite relation. The supported set-difference operations are MINUS_PRIMARY (Minus all=false) and MINUS_PRIMARY_ALL (Minus all=true); the supported intersection operations are INTERSECTION_MULTISET (Intersect all=false) and INTERSECTION_MULTISET_ALL (Intersect all=true). Closes substrait-io#303
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the two incorrect
SetOp→ Calcite relation mappings in the Substrait → Calcite converter (SubstraitRelNodeConverter):MINUS_MULTISET→ CalciteMinus(all=true)INTERSECTION_PRIMARY→ CalciteIntersect(all=false)These mappings were incorrect — the Substrait spec behaviour of these set operations does not match the Calcite relations they were mapped to. They were retained temporarily (with a
TODOreferencing this issue) to let users migrate off of them after the correct mappings were introduced.Neither
MINUS_MULTISETnorINTERSECTION_PRIMARYhas an equivalent Calcite relation, so converting a plan containing either to Calcite now throwsUnsupportedOperationException(thedefaultbranch of the switch), consistent with how the test utilities already excluded them.Note:
MINUS_MULTISETandINTERSECTION_PRIMARYremain valid Substrait spec operations — this only touches the Isthmus (Calcite) conversion. The core POJO model, dialect enum, type-derivation,SqlKindFromRel, and the Calcite → Substrait direction (SubstraitRelVisitor, which already produces the correct ops) are unchanged.Changes
MINUS_MULTISETandINTERSECTION_PRIMARYcases fromSubstraitRelNodeConverter.getRelBuilder.TODOcomment referencing the issue.UnsupportedOperationExceptionon conversion.SetUtils.Breaking change
Converting a Substrait plan that uses
MINUS_MULTISETorINTERSECTION_PRIMARYto Calcite now throwsUnsupportedOperationExceptioninstead of producing an incorrect Calcite relation. The supported set-difference operations areMINUS_PRIMARY(Minus all=false) andMINUS_PRIMARY_ALL(Minus all=true); the supported intersection operations areINTERSECTION_MULTISET(Intersect all=false) andINTERSECTION_MULTISET_ALL(Intersect all=true).Testing
./gradlew :isthmus:test --tests "io.substrait.isthmus.SubstraitRelNodeConverterTest" --tests "io.substrait.isthmus.Substrait2SqlTest" --tests "io.substrait.isthmus.ProtoPlanConverterTest"./gradlew spotlessApply :core:check :isthmus:checkCloses #303
🤖 Generated with AI