[TEST][VL] Forward Hadoop filesystem configuration per runtime

[jackylee-ch]

What changes are proposed in this pull request?

This PR forwards user-provided Hadoop filesystem configuration from Spark to each native Velox runtime.

• Collect and normalize user-provided filesystem settings once per SparkSession.
• Propagate the settings through native execution and writer paths without embedding credentials in Substrait plans.
• Create per-runtime connector configuration with credential-safe runtime identity and sensitive-setting redaction, while preserving existing APIs.

How was this patch tested?

Yes. The change was sufficiently covered by targeted Scala and C++ unit tests, Spark 3.3 and Spark 3.5 compilation, and real native Parquet write validation.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex GPT-5

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR forwards user-provided Hadoop filesystem (fs.*) configuration from Spark into each native Velox runtime in a credential-safe way, ensuring native reads/writes use the same filesystem settings as the originating SparkSession.

Changes:

• Add a HadoopConfContributor SPI and HadoopConfCollector to snapshot and cache user-provided filesystem settings per SparkSession.
• Transport the filesystem snapshot through whole-stage RDD execution and datasource writer/schema paths, and merge it into Velox runtime/connector configuration.
• Improve redaction behavior and runtime identity generation to avoid leaking or fingerprinting credentials; add Scala and C++ tests.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
shims/spark33/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala	Pass SparkSession into nativeConf so writer can collect per-session fs config.
shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala	Same session-aware nativeConf wiring for Parquet writes.
shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala	Same session-aware nativeConf wiring for ORC writes.
shims/common/src/main/scala/org/apache/gluten/execution/datasource/GlutenFormatWriterInjects.scala	Add default session-aware nativeConf overload for compatibility.
gluten-ut/test/src/test/scala/org/apache/gluten/runtime/RuntimesSuite.scala	Add tests ensuring runtime resource IDs don’t leak/fingerprint credentials.
gluten-ut/test/src/test/scala/org/apache/gluten/execution/HadoopConfTransportSuite.scala	Add tests for fsConf transport through RDDs and IteratorApi overload delegation.
gluten-ut/test/src/test/scala/org/apache/gluten/execution/HadoopConfCollectorSuite.scala	Add tests for collection/filtering/normalization/caching behavior.
gluten-ut/test/src/test/scala/org/apache/gluten/config/GlutenRuntimeConfigSuite.scala	Add tests asserting default redaction regex is injected / user override preserved.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala	Add serializable fsConf field and pass it into IteratorApi final-stage iterator creation.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala	Collect fsConf once per session and thread it into whole-stage RDD constructors.
gluten-substrait/src/main/scala/org/apache/gluten/execution/HadoopConfCollector.scala	New collector that filters user sources, normalizes keys, and caches per SparkSession.
gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala	Add serializable fsConf field and pass it into IteratorApi first-stage iterator creation.
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala	Add default native redaction regex to both session and backend native conf defaults.
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala	Add fsConf overloads with default delegation to preserve existing backends.
gluten-core/src/main/scala/org/apache/gluten/config/HadoopConfContributor.scala	New SPI for backends/components to declare fs prefixes of interest.
gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtimes.scala	Replace resource key stringification with SHA-256-based credential-safe resourceId.
cpp/velox/utils/VeloxWholeStageDumper.h	Expose formatConfigForDump helper for testing and section-specific redaction.
cpp/velox/utils/VeloxWholeStageDumper.cc	Redact backend/session conf sections with their own regex; use helper to format dump.
cpp/velox/utils/ConfigExtractor.h	Declare filesystem config merge helper and invalid-GCS sentinel constants.
cpp/velox/utils/ConfigExtractor.cc	Implement per-runtime filesystem overlay merge, validate GCS runtime config, fail closed.
cpp/velox/tests/RuntimeTest.cc	Add tests for config merging, connector overlay behavior, and dump redaction semantics.
cpp/velox/operators/writer/VeloxParquetDataSourceGCS.h	Initialize GCS filesystem with per-runtime Hive connector config instead of nullptr.
cpp/velox/compute/VeloxRuntime.cc	Register connectors with per-runtime Hive config overlay derived from runtime fsConf.
cpp/velox/compute/VeloxBackend.h	Add overloads to create connectors with explicit connector configs (preserve old overloads).
cpp/velox/compute/VeloxBackend.cc	Implement new connector factory overloads and delegate old ones to new signatures.
cpp/core/tests/GlutenConfigTest.cc	Add tests for redaction search semantics and always-redact UGI identity keys.
cpp/core/tests/CMakeLists.txt	Register new gluten_config_test target.
cpp/core/config/GlutenConfig.h	Declare redaction helpers and move required includes to header.
cpp/core/config/GlutenConfig.cc	Add shouldRedactConfigKey and switch to search semantics in printConfig.
backends-velox/src/test/scala/org/apache/spark/sql/execution/datasources/velox/VeloxHadoopConfWriterSuite.scala	Add tests for writer fsConf collection/normalization and schema inspection resource handling.
backends-velox/src/test/scala/org/apache/gluten/backendsapi/velox/VeloxHadoopConfTransportSuite.scala	Add tests for Velox Iterator extraConf merging and control-key authority.
backends-velox/src/main/scala/org/apache/spark/sql/execution/datasources/velox/VeloxParquetWriterInjects.scala	Override session-aware nativeConf to include per-session collected filesystem config.
backends-velox/src/main/scala/org/apache/spark/sql/execution/datasources/velox/VeloxFormatWriterInjects.scala	Pass runtime fsConf into runtime creation; run schema inspection under TaskResources + session.
backends-velox/src/main/scala/org/apache/gluten/datasource/VeloxDataSourceUtil.scala	Add fsConf overloads for schema inspection and robust handle/resource cleanup helper.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala	Add fsConf-aware iterator overloads and build merged extraConf including fsConf.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala	Implement HadoopConfContributor so Velox declares interested Hadoop fs prefixes.
backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenParquetFileFormat.scala	Pass SparkSession into nativeConf for Delta Parquet writes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[jackylee-ch]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86