[branch-4.0](cherry-pick) Backport Paimon and Iceberg fixes by Gabriel39 · Pull Request #65397 · apache/doris

Gabriel39 · 2026-07-08T14:50:25Z

Proposed changes

Backport the following PRs to branch-4.0 in order:

This includes task executor scan handle validation, S3 TVF regression isolation, Paimon JNI IOManager/profile improvements, and mixed-case external table column handling for Iceberg/Paimon.

Notes

PR [fix](fe) Preserve external table column name case #65094 was adapted for branch-4.0; the final JDBC catalog regression expectation commit only touched a test file that is already deleted on this branch, so it was skipped as empty after conflict resolution.
The Iceberg sort-order fragment from [fix](fe) Preserve external table column name case #65094 was not applied because SortFieldInfo and the corresponding create-table sort-order path are not present on branch-4.0.

Validation

git diff --check origin/branch-4.0..HEAD

Cherry-picked from #65054, #65147, #65332, #65354, and #65094.

Issue Number: None Related PR: None Problem Summary: Task-executor scan scheduling could pass a null or invalid task handle into TimeSharingTaskExecutor. enqueue_splits and related paths cast the base TaskHandle to TimeSharingTaskHandle and immediately dereferenced the result, so a broken ScannerContext task-handle invariant caused BE to crash with SIGSEGV instead of returning a diagnostic error. This change validates scanner context, scan task, and task handle before submitting scan splits, and validates the task handle type at TimeSharingTaskExecutor entry points before dereferencing it. None - Test: Unit Test - Added TimeSharingTaskExecutorTest coverage for null and invalid task handles. - Tried: JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home ./run-be-ut.sh --run --filter='TimeSharingTaskExecutorTest.*'; build failed during CMake configure because thirdparty/installed is missing Snappy. - Behavior changed: Yes. Invalid task-executor scan handles now return InternalError instead of dereferencing a null cast result. - Does this need documentation: No (cherry picked from commit 82cb6bf)

Issue Number: None Related PR: None Problem Summary: The S3 TVF insert regression case wrote all runs under a fixed regression/insert_tvf_test prefix while also using delete_existing_files=true and wildcard reads. Concurrent regression jobs could delete each other's directories or infer schema from another run's files, causing flaky mismatches or missing columns. This change gives each suite run a UUID-scoped S3 prefix and uses that same prefix for both writes and reads, so deletes and wildcard reads are isolated to the current run. None - Test: Regression test attempted: ./run-regression-test.sh --run -d external_table_p0/tvf/insert -s test_insert_into_s3_tvf. It could not complete because local FE at 127.0.0.1:9030 refused the MySQL connection before the first SQL statement. - Behavior changed: No, test-only path isolation. - Does this need documentation: No (cherry picked from commit 7a0116c)

Issue Number: None Related PR: None Problem Summary: Paimon JNI reads did not attach Paimon IOManager when creating TableRead, so primary-key merge reads in the Java scanner could not spill through Paimon IOManager. This change forwards catalog-level Paimon JNI IOManager options from FE to BE, creates and closes the IOManager in the Java scanner, supports an optional custom IOManager implementation class that is already available on the scanner classpath, and injects a default IOManager temp directory for both legacy and Format V2 JNI scan paths when the user enables IOManager without specifying a temp dir. The default temp directory is `${storage_root_path}/paimon_jni_scanner_io_tmp` for each BE storage root. Paimon JNI scanner can enable Paimon IOManager spill with catalog property `paimon.doris.enable_jni_io_manager=true`; `paimon.doris.jni_io_manager.tmp_dir` can optionally override the spill temp directory, and `paimon.doris.jni_io_manager.impl_class` can optionally select a custom `org.apache.paimon.disk.IOManager` implementation already available on the scanner classpath. - Test: Unit Test / Manual test - `mvn test -Dmaven.build.cache.enabled=false -pl be-java-extensions/paimon-scanner -Dtest=PaimonJniScannerTest` passed on `gabriel@10.26.20.3:/mnt/disk3/gabriel/Workspace/dev2/doris`. - `python3 build-support/run_clang_format.py ...` check-only passed for modified BE C++ files. - `git diff --check` passed. - `ninja -C be/ut_build_ASAN src/format/CMakeFiles/Format.dir/table/paimon_jni_reader.cpp.o src/format/CMakeFiles/Format.dir/__/format_v2/jni/paimon_jni_reader.cpp.o` compiled the modified reader objects on the remote validation host. - `./run-be-ut.sh --run --filter="PaimonJniReaderTest.*"` was attempted on the remote validation host but stopped before target tests due to unrelated missing header `faiss/invlists/PreadInvertedLists.h` while compiling `be/src/storage/index/ann/faiss_ann_index.cpp`. - `./run-fe-ut.sh --run org.apache.doris.datasource.paimon.source.PaimonScanNodeTest` was attempted on the remote validation host but stopped before target tests due to unrelated generated `ScalarBuiltins` compile errors referencing missing `FunctionSet`. - Behavior changed: Yes. When enabled, Paimon JNI reads now create a Paimon IOManager and use a storage-root scoped temp directory by default. Users can optionally choose a preinstalled custom IOManager implementation class. - Does this need documentation: No. The PR description documents the catalog properties, default temp directory, and custom IOManager implementation behavior. (cherry picked from commit c6bca61)

Issue Number: None Related PR: None Problem Summary: Paimon JNI scans can create Paimon SDK internal async file readers, especially for primary-key merge reads over large ORC files. Doris scanner concurrency limits do not directly expose those internal async reader threads or Java heap pressure in the query profile. This change adds lightweight profile metrics from the Paimon JNI scanner for active scanner counts, async reader thread counts, JVM heap/non-heap usage, split and predicate diagnostic sizes, async threshold configuration visibility, and readBatch/open timing counters. None - Test: Unit Test - JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dtest=PaimonJniScannerTest -DfailIfNoTests=false -Ddoris.thrift.executable=/usr/local/bin/thrift - Behavior changed: No - Does this need documentation: No (cherry picked from commit 961726d)

Issue Number: None Related PR: None Problem Summary: Paimon JNI scanner observability was only collected by the legacy JNI reader path. FileScannerV2 uses the format-v2 JNI table reader, which registered the Java getStatistics method but never called it before closing the scanner, so default v2 Paimon JNI scans missed the added profile counters. The async threshold diagnostic also read only prefixed JNI params, while normal Paimon catalog/table configuration is available from the deserialized table options. This change collects Java scanner statistics in the v2 JNI reader before scanner close and reads the async threshold from Paimon table options with the existing params fallback. None - Test: Unit Test - JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dtest=PaimonJniScannerTest -DfailIfNoTests=false -Ddoris.thrift.executable=/usr/local/bin/thrift - python3 build-support/run_clang_format.py --clang-format-executable $(brew --prefix llvm@16)/bin/clang-format --style file --inplace false be/src/format_v2/jni/jni_table_reader.cpp be/src/format_v2/jni/jni_table_reader.h - git diff --check - ./run-be-ut.sh --run --filter="PaimonJniReaderTest*" (failed before running tests: missing thirdparty/installed/bin/protoc and Snappy) - Behavior changed: No - Does this need documentation: No (cherry picked from commit 61eb6bf)

Issue Number: None Related PR: None Problem Summary: Paimon JNI scanner statistics include both scanner-local cumulative values and process-wide snapshot values such as active scanner counts, async reader thread counts, and JVM memory usage. Exporting all of them through the additive JNI profile counter path made multi-split scans multiply snapshot values by the number of collected splits. This change extends the JNI statistics profile collectors to support non-additive gauge and peak metric types, then reports process-wide snapshots as set/max metrics while keeping scanner-local rows, calls, encoded lengths, and timers additive. None - Test: Unit Test - JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dtest=PaimonJniScannerTest -DfailIfNoTests=false -Ddoris.thrift.executable=/usr/local/bin/thrift - python3 build-support/run_clang_format.py --clang-format-executable $(brew --prefix llvm@16)/bin/clang-format --style file --inplace true be/src/format/jni/jni_reader.cpp be/src/format_v2/jni/jni_table_reader.cpp be/src/format_v2/jni/jni_table_reader.h - git diff --check - Behavior changed: No - Does this need documentation: No (cherry picked from commit a08d695)

…etrics ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Paimon JNI async reader thread diagnostics counted matching threads with Thread.getAllStackTraces(), which builds stack traces for every live JVM thread even though only thread names are needed. Since scanner statistics are collected on scanner close, multi-split scans can repeat that expensive JVM-wide stack trace collection. This change uses ThreadMXBean.getAllThreadIds() and getThreadInfo(ids, 0) to inspect thread names without collecting stack traces. ### Release note None ### Check List (For Author) - Test: Unit Test - JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dtest=PaimonJniScannerTest -DfailIfNoTests=false -Ddoris.thrift.executable=/usr/local/bin/thrift - git diff --check - Behavior changed: No - Does this need documentation: No (cherry picked from commit c2b3488)

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Paimon JNI active scanner and async reader thread peak values were tracked with static JVM-lifetime high-water marks. Even with non-additive profile aggregation, later query profiles could report stale BE-wide historical peaks as if they belonged to the current query. This change removes those JVM-lifetime peak values from the per-query scanner statistics and keeps the current active scanner and async reader thread counts as snapshot gauges. ### Release note None ### Check List (For Author) - Test: Unit Test - JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dtest=PaimonJniScannerTest -DfailIfNoTests=false -Ddoris.thrift.executable=/usr/local/bin/thrift - git diff --check - Behavior changed: No - Does this need documentation: No (cherry picked from commit 9455421)

hello-stephen · 2026-07-08T14:50:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2026-07-08T14:51:02Z

run buildall

Issue Number: None Related PR: None Problem Summary: Creating Iceberg or Paimon external tables with mixed-case partition columns could fail because Doris converted top-level external column names to lower case while building external schemas and partition specs. Reading external table schemas and partition metadata also normalized some Paimon and Iceberg column names to lower case, so SHOW CREATE and partition helpers could lose the original external column spelling. This change preserves the original top-level external field names when converting Doris columns to Iceberg/Paimon schemas, resolves partition and primary key names case-insensitively back to the external canonical names, and stops schema/partition parsing paths from lowercasing external column names. Fix Iceberg and Paimon external table column name casing for mixed-case partition columns. - Test: Unit Test - Maven focused FE test: MAVEN_ARGS=-o JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl fe-core -am -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=CreateIcebergTableTest,PaimonMetadataOpsTest,IcebergUtilsTest#testParseSchemaPreservesNonLowercaseColumnNames,PaimonUtilTest#testParseSchemaPreservesNonLowercaseColumnNames - git diff --check - A broader focused run including two existing Mockito-based IcebergUtilsTest methods compiled successfully but those two methods failed locally because Mockito inline Byte Buddy could not self-attach to the Homebrew JDK 17 VM. - Behavior changed: Yes. Iceberg and Paimon external schemas, partition specs, and partition metadata now preserve external column name casing. - Does this need documentation: No (cherry picked from commit 374e8d3)

Issue Number: None Related PR: None Problem Summary: Paimon scan projection still matched Doris slot names against Paimon field names with lower-case or exact string comparisons, so mixed-case columns could be dropped from the FE projection or rejected by the JNI scanner as missing. Iceberg table creation also passed sort-order column names to the Iceberg builder without resolving them through the case-insensitive schema lookup, so ORDER BY clauses that used normalized column spelling could fail to bind to mixed-case Iceberg fields. This change resolves Paimon field indexes case-insensitively in both FE and JNI scan paths and resolves Iceberg sort-order names to the canonical schema field name before building the sort order. Fix Paimon scan projection and Iceberg sort-order handling for mixed-case external column names. - Test: Unit Test - Remote FE UT: ssh gabriel@10.26.20.3, /mnt/disk3/gabriel/Workspace/dev3/doris, MAVEN_ARGS=-o ./run-fe-ut.sh --run org.apache.doris.datasource.paimon.source.PaimonScanNodeTest#testGetPathPartitionKeysReturnsTablePartitionKeys+testSetPaimonParamsUsesOrderedPartitionKeys+testGetFieldIndexMatchesMixedCaseColumns - Maven focused Paimon JNI test: MAVEN_ARGS=-o JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl be-java-extensions/paimon-scanner -am -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=PaimonJniScannerTest#testGetFieldIndexMatchesMixedCaseColumns - Maven focused Iceberg FE test: MAVEN_ARGS=-o JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn test -pl fe-core -am -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=CreateIcebergTableTest#testSortOrderResolvesNonLowercaseColumnNamesCaseInsensitively - git diff --check - Behavior changed: Yes. Paimon scan projection and Iceberg sort-order creation now resolve mixed-case external column names case-insensitively while preserving canonical schema names. - Does this need documentation: No (cherry picked from commit ecc242b)

### What problem does this PR solve? Issue Number: None Related PR: apache#65094 Problem Summary: Regenerated the Iceberg invalid Avro column name regression output after rebuilding FE and BE and rerunning the target external Iceberg case against the initialized REST catalog. ### Release note None ### Check List (For Author) - Test: Regression test - Ran test_iceberg_invaild_avro_name on the remote validation host with FE and BE rebuilt. - Behavior changed: No - Does this need documentation: No (cherry picked from commit 2310e84)

### What problem does this PR solve? Issue Number: None Related PR: apache#65094 Problem Summary: Regenerated the Iceberg invalid Avro column name regression output with a FE rebuilt from the PR changes. The expected DESC output now preserves the original mixed-case external column name. ### Release note None ### Check List (For Author) - Test: Regression test - Rebuilt FE on the remote validation host, started a temporary FE/BE cluster from the rebuilt output, and ran test_iceberg_invaild_avro_name against it. - Behavior changed: No - Does this need documentation: No (cherry picked from commit 25bc563)

Issue Number: None Related PR: apache#65094 Problem Summary: The Paimon catalog regression expected the duplicate-column diagnostic to use a lower-case column name, but the FE now preserves the original external column case and reports the duplicated column as ID. The Paimon JDBC catalog regression also treated output from a failed optional docker probe as a container name, which caused a malformed docker cp command when the spark-iceberg container was unavailable or the current user lacked docker permission. Update the expected duplicate-column message and make optional command failures return an empty result so the existing spark-iceberg availability check can skip the environment-dependent JDBC portion correctly. None - Test: Regression test - On gabriel@10.26.20.3 under /mnt/disk3/gabriel/Workspace/dev3/doris, ran test_paimon_catalog against the rebuilt PR FE/BE with jdbcUrl pointing to 127.0.0.1:49230 and hive2HdfsPort=8320. - On gabriel@10.26.20.3 under /mnt/disk3/gabriel/Workspace/dev3/doris, ran test_paimon_jdbc_catalog against the rebuilt PR FE/BE with jdbcUrl pointing to 127.0.0.1:49230 and enableJdbcTest=true; the case detected docker permission denial and skipped the spark-iceberg-dependent section as intended. - Behavior changed: No - Does this need documentation: No (cherry picked from commit ce40d6b)

Gabriel39 · 2026-07-09T03:07:48Z

run buildall

Gabriel39 · 2026-07-09T03:28:52Z

run buildall

Gabriel39 · 2026-07-09T03:54:20Z

run buildall

hello-stephen · 2026-07-09T05:11:01Z

BE UT Coverage Report

Increment line coverage 22.50% (18/80) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.51% (19565/36561)
Line Coverage	36.58% (183071/500443)
Region Coverage	33.14% (142295/429398)
Branch Coverage	34.00% (61527/180952)

hello-stephen · 2026-07-09T08:36:20Z

BE Regression && UT Coverage Report

Increment line coverage 30.00% (24/80) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.52% (25583/35769)
Line Coverage	54.38% (271263/498786)
Region Coverage	51.91% (224684/432830)
Branch Coverage	53.37% (96798/181374)

Gabriel39 · 2026-07-09T14:56:41Z

/review

Gabriel39 · 2026-07-09T14:56:48Z

run buildall

github-actions

I found two issues that should be fixed before merge.

Critical checkpoints:

Correctness: zero-column Paimon JNI scans still parse as one empty field, so the new virtual-row handling is unreachable for BE empty projections.
CI/style: git diff --check fails on trailing whitespace in the regenerated Iceberg expected output.
Duplicate suppression: no existing inline review comments were present; TSC-2 was merged into M-1.
User focus: no additional user-provided review focus was supplied.
Validation limits: this was a static review with git diff --check and targeted code tracing; I did not run the full FE/BE test suites in this runner.

Subagent conclusions: optimizer-rewrite reported no candidates. tests-session-config proposed TSC-1, accepted as M-2, and TSC-2, merged as duplicate of M-1. Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same final ledger/comment set.

github-actions

I found two issues that should be fixed before merge.

Critical checkpoints:

Correctness: zero-column Paimon JNI scans still parse as one empty field, so the new virtual-row handling is unreachable for BE empty projections.
CI/style: git diff --check fails on trailing whitespace in the regenerated Iceberg expected output.
Duplicate suppression: no existing inline review comments were present; TSC-2 was merged into M-1.
User focus: no additional user-provided review focus was supplied.
Validation limits: this was a static review with git diff --check and targeted code tracing; I did not run the full FE/BE test suites in this runner.

Subagent conclusions: optimizer-rewrite reported no candidates. tests-session-config proposed TSC-1, accepted as M-2, and TSC-2, merged as duplicate of M-1. Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same final ledger/comment set.

github-actions · 2026-07-09T15:14:33Z

 -- !desc --
 id	int	Yes	true	\N	
-test:a1b2.raw.abc-gg-1-a	text	Yes	true	\N	
+TEST:A1B2.RAW.ABC-GG-1-A	text	Yes	true	\N	


git diff --check fails on this generated output because the changed TEST:A1B2.RAW.ABC-GG-1-A row still ends with a trailing tab. The same trailing tab is present in the second changed desc row at line 32, so CI/style checks that run the whitespace gate will reject the PR until these expected-output lines are regenerated or trimmed.

github-actions · 2026-07-09T15:14:33Z

                        appendData(i, columnValue);
                    }
                    if (rows >= batchSize) {
+                        if (fields.length == 0) {


This branch still will not run for the empty-projection case coming from BE. PaimonJniReader sends empty slot lists as required_fields="" and columns_types="", but Java String.split turns each of those into a one-element array containing "". That leaves fields.length == 1, so getProjected() tries to resolve an empty field name and fails with RequiredField not found in schema before the virtual-row path can append rows. Please normalize empty required_fields/columns_types to zero-length arrays in the constructor before parsing the types.

Gabriel39 · 2026-07-09T15:21:35Z

run buildall

hello-stephen · 2026-07-09T16:34:16Z

FE UT Coverage Report

Increment line coverage 54.55% (48/88) 🎉
Increment coverage report
Complete coverage report

Gabriel39 added 8 commits July 8, 2026 22:45

Gabriel39 requested a review from morningman as a code owner July 8, 2026 14:50

Gabriel39 added 5 commits July 9, 2026 10:37

Gabriel39 changed the title ~~[branch-4.0] Backport PR #65054 #65147 #65332 #65354~~ [branch-4.0][cherry-pick] Backport Paimon and Iceberg fixes Jul 9, 2026

[fix](ci) Fix backport formatting checks

b902b30

Gabriel39 changed the title ~~[branch-4.0][cherry-pick] Backport Paimon and Iceberg fixes~~ [branch-4.0](cherry-pick) Backport Paimon and Iceberg fixes Jul 9, 2026

[fix](paimon) Adapt backend JDBC options for branch-4.0

ca68c1b

[fix](paimon) Use branch-4.0 thrift options

4301c21

[fix](ci) Adapt branch-4.0 backport tests

e694c9f

github-actions Bot requested changes Jul 9, 2026

View reviewed changes

[fix](paimon) Handle empty JNI projections

ce8fa07

Uh oh!

Conversation

Gabriel39 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Notes

Validation

Uh oh!

hello-stephen commented Jul 8, 2026

Uh oh!

Gabriel39 commented Jul 8, 2026

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

hello-stephen commented Jul 9, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jul 9, 2026

BE Regression && UT Coverage Report

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Gabriel39 commented Jul 9, 2026

Uh oh!

hello-stephen commented Jul 9, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gabriel39 commented Jul 8, 2026 •

edited

Loading