Skip to content

Spark: Route defaultNamespace call to wrapped catalog in SparkSessionCatalog #17029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Dzeri96 wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Spark: Route defaultNamespace call to wrapped catalog in SparkSessionCatalog #17029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a2e44ca
Issue 14424: Deprecated default-namespace config. Now defaultDatabase…
Dzeri96 Jul 1, 2026
392b8b8
Merge branch 'main' into 14424-use-defaultDatabase-as-config-source
Dzeri96 Aug 4, 2026
3e250ff
Merge branch 'main' into 14424-use-defaultDatabase-as-config-source
Dzeri96 Sep 8, 2026
004a922
Issue 14424: Removed changes to SparkCatalog. Will be a different PR
Dzeri96 Sep 8, 2026
848c520
Issue 14424: Added changes to all Spark versions
Dzeri96 Sep 8, 2026
eae6f5b
Issue 14424: Fixed active spark session detection in CustomSparkTestB…
Dzeri96 Sep 9, 2026
6fabe44
Issue 14424: Removed unnecessary exception from test signature in Tes…
Dzeri96 Sep 9, 2026
cd5ce09
Issue 14424: Reworked defaultDatabase config test
Dzeri96 Sep 25, 2026
e8a1168
Issue 14424: Reverted accidentally-commited gradle.properties
Dzeri96 Sep 25, 2026
686f3aa
Issue 14424: Re-worked defaultDatabase tests with SQLConf.withExistin…
Dzeri96 Sep 25, 2026
7cc488f
Merge branch 'main' into 14424-use-defaultDatabase-as-config-source
Dzeri96 Oct 6, 2026
e541296
Issue 14424: Completely removed tests for defaultDatabase calls in Sp…
Dzeri96 Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/docs/spark-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ Both catalogs are configured using properties nested under the catalog name. Com
| spark.sql.catalog._catalog-name_.catalog-impl | | The custom Iceberg catalog implementation. If `type` is null, `catalog-impl` must not be null. |
| spark.sql.catalog._catalog-name_.io-impl | | The custom FileIO implementation. |
| spark.sql.catalog._catalog-name_.metrics-reporter-impl | | The custom MetricsReporter implementation. |
| spark.sql.catalog._catalog-name_.default-namespace | default | The default current namespace for the catalog |
| spark.sql.catalog._catalog-name_.default-namespace | default | The default current namespace for the `SparkCatalog`. |
| spark.sql.catalog._catalog-name_.defaultDatabase | default | The default current namespace for the `SparkSessionCatalog`. |
| spark.sql.catalog._catalog-name_.uri | thrift://host:port | Hive metastore URL for hive typed catalog, REST URL for REST typed catalog |
| spark.sql.catalog._catalog-name_.warehouse | hdfs://nn:8020/warehouse/path | Base path for the warehouse directory |
| spark.sql.catalog._catalog-name_.cache-enabled | `true` or `false` | Whether to enable catalog cache, default value is `true` |
Expand Down
3 changes: 1 addition & 2 deletions spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@
public class SparkSessionCatalog<
T extends TableCatalog & FunctionCatalog & SupportsNamespaces & ViewCatalog>
extends BaseCatalog implements CatalogExtension {
private static final String[] DEFAULT_NAMESPACE = new String[] {"default"};

private String catalogName = null;
private TableCatalog icebergCatalog = null;
Expand Down Expand Up @@ -92,7 +91,7 @@ protected TableCatalog buildSparkCatalog(String name, CaseInsensitiveStringMap o

@Override
public String[] defaultNamespace() {
return DEFAULT_NAMESPACE;
return getSessionCatalog().defaultNamespace();
}

@Override
Expand Down
3 changes: 1 addition & 2 deletions spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@
public class SparkSessionCatalog<
T extends TableCatalog & FunctionCatalog & SupportsNamespaces & ViewCatalog>
extends BaseCatalog implements CatalogExtension {
private static final String[] DEFAULT_NAMESPACE = new String[] {"default"};

private String catalogName = null;
private TableCatalog icebergCatalog = null;
Expand Down Expand Up @@ -93,7 +92,7 @@ protected TableCatalog buildSparkCatalog(String name, CaseInsensitiveStringMap o

@Override
public String[] defaultNamespace() {
return DEFAULT_NAMESPACE;
return getSessionCatalog().defaultNamespace();
}

@Override
Expand Down
3 changes: 1 addition & 2 deletions spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@
public class SparkSessionCatalog<
T extends TableCatalog & FunctionCatalog & SupportsNamespaces & ViewCatalog>
extends BaseCatalog implements CatalogExtension {
private static final String[] DEFAULT_NAMESPACE = new String[] {"default"};

private String catalogName = null;
private TableCatalog icebergCatalog = null;
Expand Down Expand Up @@ -94,7 +93,7 @@ protected TableCatalog buildSparkCatalog(String name, CaseInsensitiveStringMap o

@Override
public String[] defaultNamespace() {
return DEFAULT_NAMESPACE;
return getSessionCatalog().defaultNamespace();
}

@Override
Expand Down
3 changes: 1 addition & 2 deletions spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,6 @@
public class SparkSessionCatalog<
T extends TableCatalog & FunctionCatalog & SupportsNamespaces & ViewCatalog>
extends BaseCatalog implements CatalogExtension {
private static final String[] DEFAULT_NAMESPACE = new String[] {"default"};

private String catalogName = null;
private TableCatalog icebergCatalog = null;
Expand Down Expand Up @@ -98,7 +97,7 @@ protected TableCatalog buildSparkCatalog(String name, CaseInsensitiveStringMap o

@Override
public String[] defaultNamespace() {
return DEFAULT_NAMESPACE;
return getSessionCatalog().defaultNamespace();
}

@Override
Expand Down
Loading