Repository navigation
Update main PostgreSQL versions and test dependencies - #8973
ibrahim halatci (ihalatci) wants to merge 25 commits into
Conversation
Use paired the-process dev images from f73fc966f1e85a4b852d93c0702135fddb7c18da, verified by build-test-images run 43397208581. Keep snapshot, upgrade and nightly version matrices consistent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pin the corrected the-process images at 7a9bc51ca3605251111be15eb158c73e5af744ad, published by successful build-test-images run 43398988757. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Decorate injected Vars with their outer joins' nulling relids on PostgreSQL 16 and later so join removal does not invalidate catalog filters. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 7bc5ff1)
Keep recursive planning consistent with PostgreSQL join removal without bypassing managed-local routing. Preserve dropped attribute positions and cover right joins, join aliases and materialized CTEs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 75814af)
Adapt imported planner fixes to the list iteration macro used on main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the approved security updates against the metadata-only mitmproxy fork, retaining unrelated locked versions and Python categories. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Only annotate outer-join nulling relids when the predicate is attached above joins. PG19 security quals run at the base relation, where those relids are not valid. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Carry outer-join nullability through nested joins before replacing eliminated relation RTEs. Cover the nested distributed LEFT JOIN on the nullable side of a managed-local RIGHT JOIN. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adapt the output-only intent of dbb3a9d to main, retaining alternatives for earlier supported PostgreSQL minors. Verified function propagation and schema isolation against PG17.12,18.7,19rc2. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the existing distributed debug plan for versions that retain the join. The alternative removes only the three obsolete debug lines; the count remains101. Existing mixed-join tests pass on PG17.12,18.7,19rc2. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use uncrustify0.82.0 and citus_indent0.8.33 from the paired stylechecker image. The repository reindent recipe changes whitespace only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use verified exact the-process74bd9f9e4b446f9291459f49fc353633bbc805cc dev publication, generated from dependency source223abeeb16eea4e881da5e9ba9877caee4f0ec51. Stable suffix promotion remains unapproved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8973 +/- ##
==========================================
+ Coverage 88.81% 88.83% +0.02%
==========================================
Files 292 292
Lines 66278 66436 +158
Branches 8375 8402 +27
==========================================
+ Hits 58862 59016 +154
- Misses 4996 5000 +4
Partials 2420 2420 🚀 New features to boost your workflow:
|
Do not replace an original nullable relation merely because a planned subquery no longer appears in the transformed query. Preserve conversion of all matching transformed copies and add a deterministic wrong-result regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Move only DDL and data into the create schedule, retaining the numerical query assertion for regular-user and MX coverage. Adapt the fixture layout validated in 3e42d97. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… outputs function_propagation: PostgreSQL 17.12/18.7/19rc2 (CVE-2026-6460) reject ALTER TABLE table_1_for_circ_dep ADD COLUMN ... because the default of table_2_for_circ_dep uses table_1_for_circ_dep's row type, so PostgreSQL errors before Citus detects the cycle. Create the column up front and set its default with ALTER COLUMN ... SET DEFAULT instead. That creates the same function/table cycle without changing the row type, so Citus raises its circular-dependency error on old and new minors alike. isolation_citus_schema_distribute_undistribute: PostgreSQL 17.12/18.7/ 19rc2 (CVE-2026-6467) lock the schema during ALTER SCHEMA ... RENAME, so the rename waits for s1 on new minors but not on old ones. Make s1 lock the schema's pg_namespace row before s2 renames it, so the rename waits for s1 on every minor. local_dist_join_mixed: new minors remove the nullable "distributed" side of local LEFT JOIN distributed before Citus wraps "local", which changes the DEBUG output for that query. Only check its count. Remove function_propagation_0.out, isolation_citus_schema_distribute_undistribute_0.out and local_dist_join_mixed_0.out.
Adding nulling relids to the shard-hiding and Citus-object-hiding filters moved them above the outer join. As a result, a LEFT JOIN row that matched only a hidden row was removed instead of being null-extended. When the meta table is on the nullable side of an outer join, add the filter to the relation's security quals so it runs while scanning the relation. Elsewhere keep the old WHERE clause filter.
The query returns true whether or not the planner removes the outer join, and also returns true when the filter is placed above the join, so it does not test what its comment claims.
|
|
||
|
|
||
| static bool | ||
| ReplaceRemovedOuterJoinRelations(Node *node, RemovedOuterJoinContext *context) |
There was a problem hiding this comment.
PG's latest minors (18.7, 17.12, 16.16) can drop a LEFT JOIN to an unused distributed table from the planned query. Citus still plans the original query, which keeps that join, so the query errors with "recursive complex joins ...".
To fix this, the branch calls ReplaceRemovedOuterJoinRelations after PG planning. It replaces the removed distributed table with an empty subquery. It is skipped in two cases:
- The query has a modifying CTE.
- The query still references another distributed table, even inside a subquery.
For example, the current approach fixes:
SELECT count(*), (SELECT count(*) FROM dist2) FROM local l LEFT JOIN dist d USING (id)
However, these still fail:
WITH m AS (DELETE FROM tgt RETURNING id) SELECT count(*) FROM local l LEFT JOIN dist d USING (id)(case 1)SELECT count(*) FROM local l LEFT JOIN dist d USING (id) CROSS JOIN (SELECT count(*) FROM dist2) s(case 2)
Both work on 18.6, and this comment also applies to other PRs.
There was a problem hiding this comment.
Fixed in commit 48be61f. The two skips you identify Onur Tirtir (@onurctirtir) avoided the real problem, which is that the empty subquery can inherit a true inh field because ordinary table references default to inheritance-enabled in Postgres. This is fixed by ConvertRteToSubqueryWithEmptyResult() unconditionally setting inh = false. Consequently, the normalization can, and does, run for every distributed SELECT. It has also been refactored into a helper function that is called at the points where its really needed, post Postgres planning. The commit also fixed dropped-column numbering and aliasing on the replaced RTE, which involves considerably enhancing the normalization, but this is preferable to patching the targets in deparse, which would require changes to Postgres ruleutils.
22cbd2d to
5f5ab52
Compare
The normalization of eliminated outer joins unnecessarily excluded queries with modifying CTEs or citus tables. But as long as the inheritance flag of the RTE constructed by `ConvertRteToSubqueryWithEmptyResult()` is set to false, then that normalization can be unconditionally called. Also refactor the normalization into `PostPGPlanNormalization()`, enabling it to be called where it's actually needed, instead of bolted onto the top of every distributed plan creation. This required moving RTE identity assignment ahead of post-plan normalization, an acceptable tradeoff for clarity of the code. Outer join normalization is enhanced to fix dropped-column numbering and aliases - see `ConvertRemovedOuterJoinRelation()` for the rationale. Add regress tests for surviving distributed subquery and modifying CTE, the two cases identifed as broken by the previous approach. Also adds alias, whole-row and correlated-reference regress coverage. Finally, suppress minor-version-dependent DEBUG output while retaining result assertions.
5f5ab52 to
48be61f
Compare
Pair with local the-process commit 48cf5f54798db98f0ecdcb84a21269136c4a1a9c. The -dev-48cf5f5 images must be published before this branch is pushed or its CI is dispatched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Follow-up to 48be61f, which skipped Var remapping when the converted table had no dropped columns. System-cols and whole-row vars remained in an empty subquery RTE, which broke relevant SELECTs that referenced ctid or a whole-row var - see this commit's test cases for examples. The fix is straightforward - don't special-case system- or whole-row vars, always remap to they become typed NULLs.
There was a problem hiding this comment.
🟡 Changes recommended
The join walker skips eliminated relations in FULL joins reduced to LEFT joins, leaving the query tree inconsistent with planner restrictions and preserving a planning failure.
1 open issue
What changed in this PR
Updates Citus main’s PostgreSQL test versions and Python dependencies, with planner and catalog-visibility compatibility fixes for the newer PostgreSQL releases. This remains a dev-image validation change; the PR reports an unresolved PG19 isolation timeout in the GA CI run.
Changes:
- Updates PostgreSQL pins to 17.12, 18.7, and 19.0, retaining PG18 for SQL snapshots.
- Refreshes regression dependencies and their devcontainer mirrors.
- Handles eliminated outer-join relations and nullable catalog filtering, with regression and isolation-test updates.
| File | Description |
|---|---|
src/test/regress/sql/multi_mx_hide_shard_names.sql |
Tests shard hiding through outer joins. |
src/test/regress/sql/local_dist_join.sql |
Adds join-normalization and planned-subquery regressions. |
src/test/regress/sql/local_dist_join_mixed.sql |
Suppresses minor-version-dependent debug output. |
src/test/regress/sql/local_dist_join_load.sql |
Creates fixtures for join-remapping regressions. |
src/test/regress/sql/function_propagation.sql |
Adapts the circular-default dependency test. |
src/test/regress/sql/citus_depended_object.sql |
Tests catalog hiding on nullable join sides. |
src/test/regress/sql/arbitrary_configs_recurring_outer_join.sql |
Exercises aliases, dropped columns, and reference remapping. |
src/test/regress/spec/isolation_citus_schema_distribute_undistribute.spec |
Makes schema-rename waiting deterministic. |
src/test/regress/Pipfile.lock |
Locks updated dependencies and hashes. |
src/test/regress/Pipfile |
Updates test dependency constraints and fork pin. |
src/test/regress/expected/multi_mx_hide_shard_names.out |
Records outer-join shard-hiding results. |
src/test/regress/expected/multi_mx_hide_shard_names_0.out |
Updates alternate shard-hiding expectations. |
src/test/regress/expected/local_dist_join.out |
Records new join-regression results. |
src/test/regress/expected/local_dist_join_mixed.out |
Aligns expectations with suppressed debug output. |
src/test/regress/expected/local_dist_join_load.out |
Records added fixture setup. |
src/test/regress/expected/isolation_citus_schema_distribute_undistribute.out |
Records schema-lock waiting and completion. |
src/test/regress/expected/function_propagation.out |
Updates circular-dependency test expectations. |
src/test/regress/expected/citus_depended_object.out |
Records nullable catalog-hiding results. |
src/test/regress/expected/arbitrary_configs_recurring_outer_join.out |
Records reference-remapping regression results. |
src/include/distributed/query_utils.h |
Declares nullable-side detection helper. |
src/include/distributed/deparse_shard_query.h |
Exposes empty-subquery conversion helper. |
src/backend/distributed/worker/worker_shard_visibility.c |
Filters nullable catalog relations before joining. |
src/backend/distributed/utils/query_utils.c |
Detects nullable sides of outer joins. |
src/backend/distributed/utils/citus_depended_object.c |
Uses scan-level filtering for nullable catalog relations. |
src/backend/distributed/planner/distributed_planner.c |
Reconciles eliminated relations and remaps column references. |
src/backend/distributed/planner/deparse_shard_query.c |
Shares empty-subquery conversion and clears inheritance state. |
.github/workflows/nightly_cassert.yml |
Updates assertion-build and upgrade version matrices. |
.github/workflows/nightly_arm64.yml |
Updates ARM64 PostgreSQL minor versions. |
.github/workflows/build_and_test.yml |
Updates PostgreSQL, snapshot, and dev-image pins. |
.devcontainer/src/test/regress/Pipfile.lock |
Mirrors the regression dependency lock. |
.devcontainer/src/test/regress/Pipfile |
Mirrors regression dependency constraints. |
.devcontainer/requirements.txt |
Updates Black and adds pytokens. |
Review effort: Balanced
| rteIndexes = RemovedOuterJoinRelationIndexes(join->larg, query, context, | ||
| nullable || join->jointype == | ||
| JOIN_RIGHT); | ||
| rteIndexes = bms_join(rteIndexes, | ||
| RemovedOuterJoinRelationIndexes(join->rarg, query, context, | ||
| nullable || | ||
| join->jointype == JOIN_LEFT) | ||
| ); |
…joins PostgreSQL can reduce a FULL join to LEFT or RIGHT and then remove the unused distributed side. Consider both FULL join inputs when looking for eliminated relations, and add reduced-FULL regression cases.
Preserve unrelated locked versions and groups. Image requirements and the next verified development-image suffix must be paired before final CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pair dependency source 23c7ed5 with published the-process f91158460e69f3904a694dd26bea5f7473240b45 images. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DESCRIPTION: Updates PostgreSQL versions and regression-test dependencies.
Updates main's supported PostgreSQL pins to 17.12, 18.7, and 19.0 GA, retaining PostgreSQL 18.7 for SQL snapshots. Build/test and cassert version/upgrade matrices agree; the bounded ARM64 matrix remains PG17/18.
Current paired dependencies
5f24c0de819bddef69c21e190356228aab868668, suffix-dev-5f24c0d. Exact-source publication passed; all 13 required manifests and digests were independently verified. This revision corrects only the four generated-requirements headers to reference this PR and immutable source provenance; dependency bodies are byte-identical tof911584.23c7ed5e06c084cc6cc1cd03f03e5ac579ec4c44. Three base exporters usepipenv requirements; stylechecker usespipenv requirements --dev-only.5cd63296e7dab274e275553df76e9699ea220032; Tornado is 6.6.1. Regression Pipfile/lock remain the source of truth and are mirrored exactly into the devcontainer.pipenv verify,pipenv sync --dev,pip check, installed Git provenance, certificate generation, and actual mitmdump listener startup passed on Python 3.12.16.Other resolved targets remain pyasn1 0.6.5, filelock 4.2.1, certifi 2026.10.19, Werkzeug 3.1.9, Black 26.10.1, Brotli 1.2.1, msgpack 1.2.3, and urwid 4.0.7. Devcontainer Black requirements include pytokens.
Known q13/q70 wrong results: correction unpublished
The query-local relation-index collision is tracked by #8838; the existing correction is #8842. That correction and its focused regressions have been prepared and validated locally only, not pushed into this PR. No new fix PR or duplicate issue was created.
1455868468014038580795The controls and corrected runs use the same newer-normalization source base
ecec459d9a9f472268255d426c1998fe5d2d95ef. All six complete corrected fixtures passed comparison, each retaining 250 unique query markers per distributed/local side and exactly one error-free target with the required aggregate. Only the two existing unsupported-query error classes are excluded symmetrically; neither target is excluded. All six unfixed controls returned the wrong target.Local correction
5617d541c1c79b004d932b0249377e44efac2d5fchanges onlymulti_physical_planner.cand the existingrecurring_join_pushdownSQL/output: resolve preserved-side relids through each restriction's own PlannerInfo instead of a bitmap pooled across query roots. Current FULL/system/whole-row normalization is preserved. Its new regression negative control fails 57/54 to 0/0 without the C correction and passes when restored.Published compatibility work and earlier deterministic evidence
The published branch retains nullable catalog/visibility predicate handling, PG19 base-security-qual handling, transformed-relation evidence before original nullable-RTE conversion, and newer eliminated-join normalization. Directly necessary PostgreSQL row-type-default and schema-rename isolation output adaptations remain.
Earlier exact saved fixtures q33/q123/q88/q208 passed twelve complete replay combinations on PG17.12/18.7/19rc2 with unfiltered target checks: respectively 27/78.1481481481481481, 2/NULL, 20/59.5, and 2/1.5. These historical cases do not substitute for q13/q70 coverage. The q88 setup resides only in the coordinator create schedule, with its assertion retained in regular-user/MX execution; 18 role/node/version contexts passed.
Validation
Current exact-head 45005547966: SUCCESS, 136 successful jobs, two benchmarks skipped, including aggregate CI, style, snapshots, PG17/18/19 builds/regressions/isolation/vanilla, upgrades, arbitrary configurations, failure/enterprise-failure, pytest, and query-generator jobs. Packaging 45005548022 and code quality 45005545199 also passed on
e98c4a2b. Current PR checks contain no failed or pending checks.Previous dependency-head
05c3b54b44999588017: SUCCESS, 136 successful jobs, two benchmarks skipped, including aggregate CI, style, snapshots, PG17/18/19 builds/regressions/isolation/vanilla, upgrades, all arbitrary configurations, failure/enterprise-failure, pytest, and query-generator jobs. Its packaging and code quality also passed. These are historical evidence, not replacement-head clearance.For the unpublished q13/q70 correction, each actual PG17.12/18.7/19.0:
CFLAGS=-Werror ./configure --with-security-flags,make -s -j4, andmake -s install-all: passed.make -s -C src/test/regress check-custom-schedule SCHEDULE=minimal_schedule WORKERCOUNT=2 EXTRA_TESTS="recurring_join_pushdown citus_depended_object function_propagation local_dist_join_mixed local_dist_join_load local_dist_join pg18": 13/13 passed each.citus_indent src/backend/distributed/planner/multi_physical_planner.c --check --quiet: passed with tools 0.8.33 / uncrustify 0.82.0.Historical GA head
36a1c1a5bused-dev-48cf5f5: 44575297529 initially failed solely on a 360-second concurrent-distribution isolation timeout and downstream CI. The authorized failed-job-only second attempt passed; unchanged local exact-spec replay also passed. The timeout's root cause was not established. That run and earlier RC2 43424196914 are historical, not the current paired dependency gate.This remains the dev-image testing/review phase. Known q13/q70 correction publication remains separately gated; green random CI is not wrong-result clearance. No stable-suffix promotion or merges are included.