Skip to content

fix(optimizer): keep projections referenced by SORT BY / DISTRIBUTE BY / CLUSTER BY - #8302

Open
fivetran-kwoodbeck wants to merge 3 commits into
mainfrom
fix/pushdown-projections-spark-sort-distribute-cluster
Open

fix(optimizer): keep projections referenced by SORT BY / DISTRIBUTE BY / CLUSTER BY#8302
fivetran-kwoodbeck wants to merge 3 commits into
mainfrom
fix/pushdown-projections-spark-sort-distribute-cluster

Conversation

@fivetran-kwoodbeck

Copy link
Copy Markdown
Collaborator

pushdown_projections kept projections referenced by ORDER BY, but Spark's SORT BY, DISTRIBUTE BY and CLUSTER BY can reference select aliases too and were not checked, so their projections were pruned. Treat all three clauses as output references, hoisting them onto the exp.SetOperation when they trail a set operation so both arms keep the projection.

-- input (spark)
SELECT t.a FROM (SELECT a, b + c AS s FROM x SORT BY s) t

-- old output: alias `s` no longer exists -> UNRESOLVED_COLUMN
SELECT t.a AS a FROM (SELECT x.a AS a FROM x AS x SORT BY s) AS t

-- fixed output
SELECT t.a AS a FROM (SELECT x.a AS a, x.b + x.c AS s FROM x AS x SORT BY s) AS t

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

SQLGlot Integration Test Results

✅ All tests passed

Comparing:

  • this branch (sqlglot:fix/pushdown-projections-spark-sort-distribute-cluster @ sqlglot e3fce8b)
  • baseline (main @ sqlglot d7dd2cd)

Overall

main: 182937 total, 163862 passed (pass rate: 89.6%)

sqlglot:fix/pushdown-projections-spark-sort-distribute-cluster: 182937 total, 163862 passed (pass rate: 89.6%)

Transitions:
No change

✅ All tests passed

Comment thread sqlglot/optimizer/pushdown_projections.py Outdated
@fivetran-kwoodbeck
fivetran-kwoodbeck force-pushed the fix/pushdown-projections-spark-sort-distribute-cluster branch from 42a4bf5 to ccfc48c Compare September 3, 2026 19:34
Comment thread sqlglot/optimizer/pushdown_projections.py
Comment thread sqlglot/optimizer/pushdown_projections.py Outdated

@georgesittas georgesittas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once final concern(s) are addressed. @geooo109 wanna take another look?

@fivetran-kwoodbeck
fivetran-kwoodbeck force-pushed the fix/pushdown-projections-spark-sort-distribute-cluster branch from ccfc48c to 16da147 Compare September 4, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants