Skip to content

Update main PostgreSQL versions and test dependencies - #8973

Open
ibrahim halatci (ihalatci) wants to merge 25 commits into
mainfrom
ihalatci-citus-main-pg-update
Open

ibrahim halatci (ihalatci) wants to merge 25 commits into
mainfrom
ihalatci-citus-main-pg-update

Conversation

@ihalatci

@ihalatci ibrahim halatci (ihalatci) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

DESCRIPTION: Updates PostgreSQL versions and regression-test dependencies.

Replacement-image consumer gate passed; known planner wrong results remain unresolved in the published branch. Exact remote head e98c4a2bba1d2562b27f598868210f6235021499 consumes -dev-5f24c0d. Full paired Build & Test passed 136 jobs, with two benchmarks skipped. Packaging and code quality passed on that same head. The previous 05c3b54b gate is historical. Random query-generator success does not clear the exact q13/q70 failures below. No merge or stable promotion.

Updates main's supported PostgreSQL pins to 17.12, 18.7, and 19.0 GA, retaining PostgreSQL 18.7 for SQL snapshots. Build/test and cassert version/upgrade matrices agree; the bounded ARM64 matrix remains PG17/18.

Current paired dependencies

  • Image PR: citusdata/the-process#267. Exact image source 5f24c0de819bddef69c21e190356228aab868668, suffix -dev-5f24c0d. Exact-source publication passed; all 13 required manifests and digests were independently verified. This revision corrects only the four generated-requirements headers to reference this PR and immutable source provenance; dependency bodies are byte-identical to f911584.
  • Image requirements derive from published dependency-only Citus source 23c7ed5e06c084cc6cc1cd03f03e5ac579ec4c44. Three base exporters use pipenv requirements; stylechecker uses pipenv requirements --dev-only.
  • citusdata/mitmproxy#6 is merged. The exact merged PR head is pinned at 5cd63296e7dab274e275553df76e9699ea220032; Tornado is 6.6.1. Regression Pipfile/lock remain the source of truth and are mirrored exactly into the devcontainer.
  • This follow-up preserves unrelated lock entries, versions, package groups, and markers. Standard Pipenv resolution generated target hashes; final pipenv verify, pipenv sync --dev, pip check, installed Git provenance, certificate generation, and actual mitmdump listener startup passed on Python 3.12.16.

Other resolved targets remain pyasn1 0.6.5, filelock 4.2.1, certifi 2026.10.19, Werkzeug 3.1.9, Black 26.10.1, Brotli 1.2.1, msgpack 1.2.3, and urwid 4.0.7. Devcontainer Black requirements include pytokens.

Known q13/q70 wrong results: correction unpublished

The query-local relation-index collision is tracked by #8838; the existing correction is #8842. That correction and its focused regressions have been prepared and validated locally only, not pushed into this PR. No new fix PR or duplicate issue was created.

Exact saved fixture Published-source control on PG17.12/18.7/19.0 Locally corrected result on all three
q13, artifact 14558684680 count 0 / avg NULL count 6 / avg 0.50000000000000000000
q70, original artifact 14038580795 count 0 / avg NULL count 26 / avg 18.3333333333333333

The controls and corrected runs use the same newer-normalization source base ecec459d9a9f472268255d426c1998fe5d2d95ef. All six complete corrected fixtures passed comparison, each retaining 250 unique query markers per distributed/local side and exactly one error-free target with the required aggregate. Only the two existing unsupported-query error classes are excluded symmetrically; neither target is excluded. All six unfixed controls returned the wrong target.

Local correction 5617d541c1c79b004d932b0249377e44efac2d5f changes only multi_physical_planner.c and the existing recurring_join_pushdown SQL/output: resolve preserved-side relids through each restriction's own PlannerInfo instead of a bitmap pooled across query roots. Current FULL/system/whole-row normalization is preserved. Its new regression negative control fails 57/54 to 0/0 without the C correction and passes when restored.

Published compatibility work and earlier deterministic evidence

The published branch retains nullable catalog/visibility predicate handling, PG19 base-security-qual handling, transformed-relation evidence before original nullable-RTE conversion, and newer eliminated-join normalization. Directly necessary PostgreSQL row-type-default and schema-rename isolation output adaptations remain.

Earlier exact saved fixtures q33/q123/q88/q208 passed twelve complete replay combinations on PG17.12/18.7/19rc2 with unfiltered target checks: respectively 27/78.1481481481481481, 2/NULL, 20/59.5, and 2/1.5. These historical cases do not substitute for q13/q70 coverage. The q88 setup resides only in the coordinator create schedule, with its assertion retained in regular-user/MX execution; 18 role/node/version contexts passed.

Validation

Current exact-head 45005547966: SUCCESS, 136 successful jobs, two benchmarks skipped, including aggregate CI, style, snapshots, PG17/18/19 builds/regressions/isolation/vanilla, upgrades, arbitrary configurations, failure/enterprise-failure, pytest, and query-generator jobs. Packaging 45005548022 and code quality 45005545199 also passed on e98c4a2b. Current PR checks contain no failed or pending checks.

Previous dependency-head 05c3b54b 44999588017: SUCCESS, 136 successful jobs, two benchmarks skipped, including aggregate CI, style, snapshots, PG17/18/19 builds/regressions/isolation/vanilla, upgrades, all arbitrary configurations, failure/enterprise-failure, pytest, and query-generator jobs. Its packaging and code quality also passed. These are historical evidence, not replacement-head clearance.

For the unpublished q13/q70 correction, each actual PG17.12/18.7/19.0:

  • CFLAGS=-Werror ./configure --with-security-flags, make -s -j4, and make -s install-all: passed.
  • Exact retained 250-query q13/q70 workloads in one backend per fixture, followed by in-place undistribution and the same local queries: six corrected comparisons and raw target assertions passed; six unfixed controls failed.
  • make -s -C src/test/regress check-custom-schedule SCHEDULE=minimal_schedule WORKERCOUNT=2 EXTRA_TESTS="recurring_join_pushdown citus_depended_object function_propagation local_dist_join_mixed local_dist_join_load local_dist_join pg18": 13/13 passed each.
  • citus_indent src/backend/distributed/planner/multi_physical_planner.c --check --quiet: passed with tools 0.8.33 / uncrustify 0.82.0.

Historical GA head 36a1c1a5b used -dev-48cf5f5: 44575297529 initially failed solely on a 360-second concurrent-distribution isolation timeout and downstream CI. The authorized failed-job-only second attempt passed; unchanged local exact-spec replay also passed. The timeout's root cause was not established. That run and earlier RC2 43424196914 are historical, not the current paired dependency gate.

This remains the dev-image testing/review phase. Known q13/q70 correction publication remains separately gated; green random CI is not wrong-result clearance. No stable-suffix promotion or merges are included.

Use paired the-process dev images from f73fc966f1e85a4b852d93c0702135fddb7c18da, verified by build-test-images run 43397208581. Keep snapshot, upgrade and nightly version matrices consistent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pin the corrected the-process images at 7a9bc51ca3605251111be15eb158c73e5af744ad, published by successful build-test-images run 43398988757.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Decorate injected Vars with their outer joins' nulling relids on PostgreSQL 16 and later so join removal does not invalidate catalog filters.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 7bc5ff1)
Keep recursive planning consistent with PostgreSQL join removal without bypassing managed-local routing. Preserve dropped attribute positions and cover right joins, join aliases and materialized CTEs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 75814af)
Adapt imported planner fixes to the list iteration macro used on main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the approved security updates against the metadata-only mitmproxy fork, retaining unrelated locked versions and Python categories.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Only annotate outer-join nulling relids when the predicate is attached above joins. PG19 security quals run at the base relation, where those relids are not valid.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Carry outer-join nullability through nested joins before replacing eliminated relation RTEs. Cover the nested distributed LEFT JOIN on the nullable side of a managed-local RIGHT JOIN.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adapt the output-only intent of dbb3a9d to main, retaining alternatives for earlier supported PostgreSQL minors. Verified function propagation and schema isolation against PG17.12,18.7,19rc2.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the existing distributed debug plan for versions that retain the join. The alternative removes only the three obsolete debug lines; the count remains101. Existing mixed-join tests pass on PG17.12,18.7,19rc2.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use uncrustify0.82.0 and citus_indent0.8.33 from the paired stylechecker image. The repository reindent recipe changes whitespace only.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use verified exact the-process74bd9f9e4b446f9291459f49fc353633bbc805cc dev publication, generated from dependency source223abeeb16eea4e881da5e9ba9877caee4f0ec51. Stable suffix promotion remains unapproved.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.15951% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.83%. Comparing base (a5635de) to head (e98c4a2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8973      +/-   ##
==========================================
+ Coverage   88.81%   88.83%   +0.02%     
==========================================
  Files         292      292              
  Lines       66278    66436     +158     
  Branches     8375     8402      +27     
==========================================
+ Hits        58862    59016     +154     
- Misses       4996     5000       +4     
  Partials     2420     2420              
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Do not replace an original nullable relation merely because a planned subquery no longer appears in the transformed query. Preserve conversion of all matching transformed copies and add a deterministic wrong-result regression.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move only DDL and data into the create schedule, retaining the numerical query assertion for regular-user and MX coverage. Adapt the fixture layout validated in 3e42d97.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… outputs

function_propagation: PostgreSQL 17.12/18.7/19rc2 (CVE-2026-6460) reject
ALTER TABLE table_1_for_circ_dep ADD COLUMN ... because the default of
table_2_for_circ_dep uses table_1_for_circ_dep's row type, so PostgreSQL
errors before Citus detects the cycle. Create the column up front and set
its default with ALTER COLUMN ... SET DEFAULT instead. That creates the
same function/table cycle without changing the row type, so Citus raises
its circular-dependency error on old and new minors alike.

isolation_citus_schema_distribute_undistribute: PostgreSQL 17.12/18.7/
19rc2 (CVE-2026-6467) lock the schema during ALTER SCHEMA ... RENAME, so the
rename waits for s1 on new minors but not on old ones. Make s1 lock the
schema's pg_namespace row before s2 renames it, so the rename waits for
s1 on every minor.

local_dist_join_mixed: new minors remove the nullable "distributed" side of
local LEFT JOIN distributed before Citus wraps "local", which changes the
DEBUG output for that query. Only check its count.

Remove function_propagation_0.out,
isolation_citus_schema_distribute_undistribute_0.out and
local_dist_join_mixed_0.out.
Adding nulling relids to the shard-hiding and Citus-object-hiding filters
moved them above the outer join. As a result, a LEFT JOIN row that matched
only a hidden row was removed instead of being null-extended.

When the meta table is on the nullable side of an outer join, add the
filter to the relation's security quals so it runs while scanning the
relation. Elsewhere keep the old WHERE clause filter.
The query returns true whether or not the planner removes the outer join, and also returns true when the filter is placed above the join, so it does not test what its comment claims.


static bool
ReplaceRemovedOuterJoinRelations(Node *node, RemovedOuterJoinContext *context)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PG's latest minors (18.7, 17.12, 16.16) can drop a LEFT JOIN to an unused distributed table from the planned query. Citus still plans the original query, which keeps that join, so the query errors with "recursive complex joins ...".

To fix this, the branch calls ReplaceRemovedOuterJoinRelations after PG planning. It replaces the removed distributed table with an empty subquery. It is skipped in two cases:

  1. The query has a modifying CTE.
  2. The query still references another distributed table, even inside a subquery.

For example, the current approach fixes:
SELECT count(*), (SELECT count(*) FROM dist2) FROM local l LEFT JOIN dist d USING (id)


However, these still fail:

  • WITH m AS (DELETE FROM tgt RETURNING id) SELECT count(*) FROM local l LEFT JOIN dist d USING (id) (case 1)
  • SELECT count(*) FROM local l LEFT JOIN dist d USING (id) CROSS JOIN (SELECT count(*) FROM dist2) s (case 2)

Both work on 18.6, and this comment also applies to other PRs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 48be61f. The two skips you identify Onur Tirtir (@onurctirtir) avoided the real problem, which is that the empty subquery can inherit a true inh field because ordinary table references default to inheritance-enabled in Postgres. This is fixed by ConvertRteToSubqueryWithEmptyResult() unconditionally setting inh = false. Consequently, the normalization can, and does, run for every distributed SELECT. It has also been refactored into a helper function that is called at the points where its really needed, post Postgres planning. The commit also fixed dropped-column numbering and aliasing on the replaced RTE, which involves considerably enhancing the normalization, but this is preferable to patching the targets in deparse, which would require changes to Postgres ruleutils.

@colm-mchugh
Colm (colm-mchugh) force-pushed the ihalatci-citus-main-pg-update branch 2 times, most recently from 22cbd2d to 5f5ab52 Compare November 18, 2026 18:22
The normalization of eliminated outer joins unnecessarily excluded queries
with modifying CTEs or citus tables. But as long as the inheritance flag of
the RTE constructed by `ConvertRteToSubqueryWithEmptyResult()` is set to
false, then that normalization can be unconditionally called. Also
refactor the normalization into `PostPGPlanNormalization()`, enabling it
to be called where it's actually needed, instead of bolted onto the top
of every distributed plan creation. This required moving RTE identity
assignment ahead of post-plan normalization, an acceptable tradeoff for
clarity of the code.

Outer join normalization is enhanced to fix dropped-column numbering
and aliases - see `ConvertRemovedOuterJoinRelation()` for the rationale.

Add regress tests for surviving distributed subquery and modifying CTE,
the two cases identifed as broken by the previous approach. Also adds
alias, whole-row and correlated-reference regress coverage. Finally,
suppress minor-version-dependent DEBUG output while retaining result
assertions.
@colm-mchugh
Colm (colm-mchugh) force-pushed the ihalatci-citus-main-pg-update branch from 5f5ab52 to 48be61f Compare November 18, 2026 19:30
Pair with local the-process commit 48cf5f54798db98f0ecdcb84a21269136c4a1a9c. The -dev-48cf5f5 images must be published before this branch is pushed or its CI is dispatched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Follow-up to 48be61f, which skipped Var remapping when the converted
table had no dropped columns. System-cols and whole-row vars remained
in an empty subquery RTE, which broke relevant SELECTs that referenced
ctid or a whole-row var - see this commit's test cases for examples.
The fix is straightforward - don't special-case system- or whole-row
vars, always remap to they become typed NULLs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The join walker skips eliminated relations in FULL joins reduced to LEFT joins, leaving the query tree inconsistent with planner restrictions and preserving a planning failure.

1 open issue
What changed in this PR

Updates Citus main’s PostgreSQL test versions and Python dependencies, with planner and catalog-visibility compatibility fixes for the newer PostgreSQL releases. This remains a dev-image validation change; the PR reports an unresolved PG19 isolation timeout in the GA CI run.

Changes:

  • Updates PostgreSQL pins to 17.12, 18.7, and 19.0, retaining PG18 for SQL snapshots.
  • Refreshes regression dependencies and their devcontainer mirrors.
  • Handles eliminated outer-join relations and nullable catalog filtering, with regression and isolation-test updates.
File Description
src/​test/​regress/​sql/​multi_mx_hide_shard_names.sql Tests shard hiding through outer joins.
src/​test/​regress/​sql/​local_dist_join.sql Adds join-normalization and planned-subquery regressions.
src/​test/​regress/​sql/​local_dist_join_mixed.sql Suppresses minor-version-dependent debug output.
src/​test/​regress/​sql/​local_dist_join_load.sql Creates fixtures for join-remapping regressions.
src/​test/​regress/​sql/​function_propagation.sql Adapts the circular-default dependency test.
src/​test/​regress/​sql/​citus_depended_object.sql Tests catalog hiding on nullable join sides.
src/​test/​regress/​sql/​arbitrary_configs_recurring_outer_join.sql Exercises aliases, dropped columns, and reference remapping.
src/​test/​regress/​spec/​isolation_citus_schema_distribute_undistribute.spec Makes schema-rename waiting deterministic.
src/​test/​regress/​Pipfile.lock Locks updated dependencies and hashes.
src/​test/​regress/​Pipfile Updates test dependency constraints and fork pin.
src/​test/​regress/​expected/​multi_mx_hide_shard_names.out Records outer-join shard-hiding results.
src/​test/​regress/​expected/​multi_mx_hide_shard_names_0.out Updates alternate shard-hiding expectations.
src/​test/​regress/​expected/​local_dist_join.out Records new join-regression results.
src/​test/​regress/​expected/​local_dist_join_mixed.out Aligns expectations with suppressed debug output.
src/​test/​regress/​expected/​local_dist_join_load.out Records added fixture setup.
src/​test/​regress/​expected/​isolation_citus_schema_distribute_undistribute.out Records schema-lock waiting and completion.
src/​test/​regress/​expected/​function_propagation.out Updates circular-dependency test expectations.
src/​test/​regress/​expected/​citus_depended_object.out Records nullable catalog-hiding results.
src/​test/​regress/​expected/​arbitrary_configs_recurring_outer_join.out Records reference-remapping regression results.
src/​include/​distributed/​query_utils.h Declares nullable-side detection helper.
src/​include/​distributed/​deparse_shard_query.h Exposes empty-subquery conversion helper.
src/​backend/​distributed/​worker/​worker_shard_visibility.c Filters nullable catalog relations before joining.
src/​backend/​distributed/​utils/​query_utils.c Detects nullable sides of outer joins.
src/​backend/​distributed/​utils/​citus_depended_object.c Uses scan-level filtering for nullable catalog relations.
src/​backend/​distributed/​planner/​distributed_planner.c Reconciles eliminated relations and remaps column references.
src/​backend/​distributed/​planner/​deparse_shard_query.c Shares empty-subquery conversion and clears inheritance state.
.github/​workflows/​nightly_cassert.yml Updates assertion-build and upgrade version matrices.
.github/​workflows/​nightly_arm64.yml Updates ARM64 PostgreSQL minor versions.
.github/​workflows/​build_and_test.yml Updates PostgreSQL, snapshot, and dev-image pins.
.devcontainer/​src/​test/​regress/​Pipfile.lock Mirrors the regression dependency lock.
.devcontainer/​src/​test/​regress/​Pipfile Mirrors regression dependency constraints.
.devcontainer/​requirements.txt Updates Black and adds pytokens.

Review effort: Balanced

Comment on lines +3288 to +3295
rteIndexes = RemovedOuterJoinRelationIndexes(join->larg, query, context,
nullable || join->jointype ==
JOIN_RIGHT);
rteIndexes = bms_join(rteIndexes,
RemovedOuterJoinRelationIndexes(join->rarg, query, context,
nullable ||
join->jointype == JOIN_LEFT)
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed by commit ecec459

Colm (colm-mchugh) and others added 4 commits November 20, 2026 20:41
…joins

PostgreSQL can reduce a FULL join to LEFT or RIGHT and then remove the
unused distributed side. Consider both FULL join inputs when looking for
eliminated relations, and add reduced-FULL regression cases.
Preserve unrelated locked versions and groups. Image requirements and the next verified development-image suffix must be paired before final CI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pair dependency source 23c7ed5 with published the-process f91158460e69f3904a694dd26bea5f7473240b45 images.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants