Skip to content

fix: return flat namespace properties from BigQueryMetastoreCatalog.load_namespace_properties#3650

Open
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:fix/bigquery-load-namespace-properties-shape
Open

fix: return flat namespace properties from BigQueryMetastoreCatalog.load_namespace_properties#3650
anxkhn wants to merge 1 commit into
apache:mainfrom
anxkhn:fix/bigquery-load-namespace-properties-shape

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
# Rationale for this change

`BigQueryMetastoreCatalog.create_namespace` stores user-supplied namespace
properties in the `parameters` map of `ExternalCatalogDatasetOptions`
(via `_create_external_catalog_dataset_options`). But
`load_namespace_properties` returned
`dataset.external_catalog_dataset_options.to_api_repr()`, which is the nested
BigQuery REST representation: it buries the properties under a `"parameters"`
key and adds a `"defaultStorageLocationUri"` key.

As a result, a simple round-trip does not behave like the other catalogs:

```python
catalog.create_namespace(("db",), {"owner": "alice"})
catalog.load_namespace_properties(("db",))["owner"]  # KeyError
# actually returns {"defaultStorageLocationUri": "...", "parameters": {"owner": "alice"}}

The base MetastoreCatalog contract types this method as -> Properties (a flat
Dict[str, str]), and the Glue and DynamoDB catalogs both return the flat map
(dict(database.get("Parameters", {}))). BigQuery was the outlier. This changes
it to return the flat parameters map, using the same .parameters accessor
already used on the table path in this file. The parameters field is None
when unset, so it is guarded to yield an empty dict for a namespace with no
properties.

Are these changes tested?

Yes. Two regression tests were added to
tests/catalog/test_bigquery_metastore.py:

  • test_load_namespace_properties_returns_flat_parameters: creates a namespace
    with {"owner": ..., "comment": ...}, then asserts load_namespace_properties
    returns exactly that flat map (the mock feeds the persisted Dataset back
    through get_dataset). This fails on the old to_api_repr() code (returns the
    nested dict) and passes with the fix.
  • test_load_namespace_properties_without_parameters: asserts a dataset whose
    options carry no parameters yields {} rather than None or a nested dict.

python -m pytest tests/catalog/test_bigquery_metastore.py -> 7 passed. Lint
(ruff, ruff-format, mypy, pydocstyle, codespell) passes on both files.
The BigQuery integration tests need Docker + Spark, which were not available in
this environment; the behavior is covered by the unit tests above.

Are there any user-facing changes?

Yes. BigQueryMetastoreCatalog.load_namespace_properties now returns the flat
namespace property map, consistent with the other catalogs, instead of the nested
BigQuery API representation. Code that relied on the previous nested shape
(reading ["parameters"][...] or ["defaultStorageLocationUri"]) would need to
read the properties directly; the previous shape did not match the documented
Properties return type.

…oad_namespace_properties

BigQueryMetastoreCatalog.create_namespace stores user properties in the
ExternalCatalogDatasetOptions.parameters map, but load_namespace_properties
returned external_catalog_dataset_options.to_api_repr(). That API
representation is a nested dict that buries the properties under a
"parameters" key and adds "defaultStorageLocationUri", so a round-trip of
create_namespace(ns, {"owner": "..."}) followed by
load_namespace_properties(ns)["owner"] raised KeyError.

Return the flat parameters map instead, matching the base MetastoreCatalog
contract and the Glue and DynamoDB catalogs. Guard the None case (parameters
is unset by default) so a namespace with no properties yields an empty dict.

Add regression tests covering both the populated round-trip and the
no-parameters case.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant