diff --git a/docs/src/glue.md b/docs/src/glue.md index f7c0c92..aee4da6 100644 --- a/docs/src/glue.md +++ b/docs/src/glue.md @@ -6,18 +6,19 @@ implements the Lance Namespace client spec. ## Background AWS Glue Data Catalog is a fully managed metadata repository that stores structural and operational metadata for data assets. -It is compatible with the Apache Hive Metastore API and can be used as a central metadata repository for data lakes. +It is based on the Apache Hive Metastore API, but uses JSON RPC instead of Apache Thrift for request response. +It can be used as a central metadata repository for data lakes. For details on AWS Glue, see the [AWS Glue Data Catalog Documentation](https://docs.aws.amazon.com/glue/latest/dg/manage-catalog.html). ## Namespace Implementation Configuration Properties The Lance Glue namespace implementation accepts the following configuration properties: -The **catalog_id** property is optional and specifies the Catalog ID of the Glue catalog (defaults to AWS account ID). +The **catalog_id** property is optional and specifies the Catalog ID of the Glue catalog to use as the starting point. When not specified, it is resolved to the caller's AWS account ID. The **endpoint** property is optional and specifies a custom Glue service endpoint for API compatible metastores. -The **region** property is optional and specifies the AWS region for all Glue operations. +The **region** property is optional and specifies the AWS region for all Glue operations. When not specified, it is resolved to the default AWS region in the caller's environment. The **access_key_id** property is optional and specifies the AWS access key ID for static credentials. @@ -51,7 +52,7 @@ The Glue namespace supports multiple authentication methods: AWS Glue Data Catalog supports a recursive catalog structure through the [GetCatalog](https://docs.aws.amazon.com/glue/latest/webapi/API_GetCatalog.html) and [GetCatalogs](https://docs.aws.amazon.com/glue/latest/webapi/API_GetCatalogs.html) APIs. This allows for multi-level namespace hierarchies. -The **root namespace** is represented by the default AWS Glue Data Catalog, which has a catalog ID of `None` or equal to the caller's AWS account ID. +The **root namespace** is represented by the default AWS Glue Data Catalog. When the `catalog_id` property is not specified or set to `None`, it is resolved to the caller's AWS account ID. Users can specify a different `catalog_id` to use another AWS account's Glue catalog as the starting point. A **child catalog** within the root catalog forms a child namespace. The [GetCatalogs](https://docs.aws.amazon.com/glue/latest/webapi/API_GetCatalogs.html) API supports `ParentCatalogId` parameter to traverse the catalog hierarchy. diff --git a/docs/src/gravitino.md b/docs/src/gravitino.md index 5744510..585f2ac 100644 --- a/docs/src/gravitino.md +++ b/docs/src/gravitino.md @@ -1,4 +1,4 @@ -# Lance Gravitino Namespace +# Apache Gravitino **Apache Gravitino** is a high-performance, geo-distributed, and federated metadata lake. It manages metadata directly in different sources, types, and regions, providing unified metadata access for data and AI assets. diff --git a/docs/src/hive2.md b/docs/src/hive2.md index cced4c5..0b116c5 100644 --- a/docs/src/hive2.md +++ b/docs/src/hive2.md @@ -1,4 +1,4 @@ -# Lance Hive 2.x Namespace Implementation Spec +# Apache Hive 2.X MetaStore Lance Namespace Implementation Spec This document describes how the Hive 2.x MetaStore implements the Lance Namespace client spec. @@ -14,8 +14,6 @@ The **client.pool-size** property is optional and specifies the size of the HMS The **root** property is optional and specifies the storage root location of the lakehouse on Hive catalog. Default value is the current working directory. -The **storage.*** prefix properties are optional and specify additional storage configurations to access tables (e.g., `storage.region=us-west-2`). - ## Object Mapping ### Namespace @@ -56,7 +54,9 @@ The implementation: **Error Handling:** -If the namespace already exists and mode is CREATE, return error code `2` (NamespaceAlreadyExists). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace already exists and mode is CREATE, return error code `2` (NamespaceAlreadyExists). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### ListNamespaces @@ -83,7 +83,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### DropNamespace @@ -113,13 +115,17 @@ The implementation: 1. Parse the table identifier to extract database and table name 2. Verify the parent namespace exists 3. Create an HMS Table object with `tableType=EXTERNAL_TABLE` -4. Set the storage descriptor with the specified or default location +4. Set the storage descriptor with the specified or default location. When location is not specified, it defaults to `{root}/{database}.db/{table}` 5. Add `table_type=lance` to the table parameters 6. Register the table in HMS **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the table already exists, return error code `5` (TableAlreadyExists). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the table already exists, return error code `5` (TableAlreadyExists). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### ListTables @@ -135,18 +141,20 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### DescribeTable -Retrieves metadata for a Lance table. Only `load_detailed_metadata=false` is supported. When `load_detailed_metadata=false`, only the table location and storage_options are returned; other fields (version, table_uri, schema, stats) are null. +Retrieves metadata for a Lance table. Only `load_detailed_metadata=false` is supported. When `load_detailed_metadata=false`, only the table location is returned; other fields (version, table_uri, schema, stats) are null. The implementation: 1. Parse the table identifier 2. Retrieve the Table object from HMS 3. Validate that it is a Lance table (check `table_type=lance`) -4. Return the table location from `storageDescriptor.location` and storage_options from `parameters` +4. Return the table location from `storageDescriptor.location` **Error Handling:** @@ -168,4 +176,8 @@ The implementation: **Error Handling:** -If the table does not exist, return error code `4` (TableNotFound). If the table is not a Lance table, return error code `13` (InvalidInput). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the table does not exist, return error code `4` (TableNotFound). + +If the table is not a Lance table, return error code `13` (InvalidInput). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). diff --git a/docs/src/hive3.md b/docs/src/hive3.md index 99f5746..382acca 100644 --- a/docs/src/hive3.md +++ b/docs/src/hive3.md @@ -1,4 +1,4 @@ -# Lance Hive 3.x Namespace Implementation Spec +# Apache Hive 3+.X MetaStore Lance Namespace Implementation Spec This document describes how the Hive 3.x MetaStore implements the Lance Namespace client spec. @@ -8,14 +8,12 @@ Apache Hive MetaStore (HMS) is a centralized metadata repository for Apache Hive ## Namespace Implementation Configuration Properties -The Lance Hive 3.x namespace implementation accepts the following configuration properties: +The Lance Hive 3+.x namespace implementation accepts the following configuration properties: The **client.pool-size** property is optional and specifies the size of the HMS client connection pool. Default value is `3`. The **root** property is optional and specifies the storage root location of the lakehouse on Hive catalog. Default value is the current working directory. -The **storage.*** prefix properties are optional and specify additional storage configurations to access tables (e.g., `storage.region=us-west-2`). - ## Object Mapping ### Namespace @@ -57,7 +55,11 @@ The implementation: **Error Handling:** -If the namespace already exists and mode is CREATE, return error code `2` (NamespaceAlreadyExists). If the parent catalog does not exist when creating a database, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace already exists and mode is CREATE, return error code `2` (NamespaceAlreadyExists). + +If the parent catalog does not exist when creating a database, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### ListNamespaces @@ -72,7 +74,9 @@ The implementation: **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### DescribeNamespace @@ -86,7 +90,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### DropNamespace @@ -116,13 +122,17 @@ The implementation: 1. Parse the table identifier to extract catalog, database, and table name 2. Verify the parent namespace exists 3. Create an HMS Table object with `tableType=EXTERNAL_TABLE` -4. Set the storage descriptor with the specified or default location +4. Set the storage descriptor with the specified or default location. When location is not specified, it defaults to `{root}/{database}.db/{table}` for the default `hive` catalog (hive2-compatible), or `{root}/{catalog}/{database}.db/{table}` for other catalogs 5. Add `table_type=lance` to the table parameters 6. Register the table in HMS **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the table already exists, return error code `5` (TableAlreadyExists). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the table already exists, return error code `5` (TableAlreadyExists). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### ListTables @@ -138,18 +148,20 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). ### DescribeTable -Retrieves metadata for a Lance table. Only `load_detailed_metadata=false` is supported. When `load_detailed_metadata=false`, only the table location and storage_options are returned; other fields (version, table_uri, schema, stats) are null. +Retrieves metadata for a Lance table. Only `load_detailed_metadata=false` is supported. When `load_detailed_metadata=false`, only the table location is returned; other fields (version, table_uri, schema, stats) are null. The implementation: 1. Parse the table identifier 2. Retrieve the Table object from HMS 3. Validate that it is a Lance table (check `table_type=lance`) -4. Return the table location from `storageDescriptor.location` and storage_options from `parameters` +4. Return the table location from `storageDescriptor.location` **Error Handling:** @@ -171,4 +183,8 @@ The implementation: **Error Handling:** -If the table does not exist, return error code `4` (TableNotFound). If the table is not a Lance table, return error code `13` (InvalidInput). If the HMS connection fails, return error code `17` (ServiceUnavailable). +If the table does not exist, return error code `4` (TableNotFound). + +If the table is not a Lance table, return error code `13` (InvalidInput). + +If the HMS connection fails, return error code `17` (ServiceUnavailable). diff --git a/docs/src/iceberg.md b/docs/src/iceberg.md index 9550ee5..54904d1 100644 --- a/docs/src/iceberg.md +++ b/docs/src/iceberg.md @@ -1,4 +1,4 @@ -# Lance Iceberg REST Catalog Implementation Spec +# Apache Iceberg REST Catalog Lance Namespace Implementation Spec This document describes how the Apache Iceberg REST Catalog implements the Lance Namespace client spec. @@ -10,26 +10,31 @@ Apache Iceberg REST Catalog is a standardized REST API for interacting with Iceb The Lance Iceberg REST Catalog namespace implementation accepts the following configuration properties: -| Property | Required | Default | Description | -|----------|----------|---------|-------------| -| `endpoint` | Yes | - | Iceberg REST Catalog server endpoint URL (e.g., `http://localhost:8181`). Must start with `http://` or `https://`. | -| `warehouse` | No | - | Warehouse identifier. Some Iceberg REST implementations require this. The warehouse name is resolved to an API prefix via the `/v1/config` endpoint. | -| `auth_token` | No | - | Bearer token for authentication. | -| `credential` | No | - | OAuth2 client credential in `client_id:client_secret` format for client credentials authentication flow. | -| `connect_timeout` | No | `10000` | Connection timeout in milliseconds. | -| `read_timeout` | No | `30000` | Read timeout in milliseconds. | -| `max_retries` | No | `3` | Maximum number of retries for failed requests. | -| `root` | No | current working directory | Default storage root location for tables. | +The **endpoint** property is required and specifies the Iceberg REST Catalog server endpoint URL (e.g., `http://localhost:8181`). Must start with `http://` or `https://`. + +The **auth_token** property is optional and specifies the bearer token for authentication. + +The **credential** property is optional and specifies the OAuth2 client credential in `client_id:client_secret` format for client credentials authentication flow. + +The **connect_timeout** property is optional and specifies the connection timeout in milliseconds (default: 10000). + +The **read_timeout** property is optional and specifies the read timeout in milliseconds (default: 30000). + +The **max_retries** property is optional and specifies the maximum number of retries for failed requests (default: 3). + +The **root** property is optional and specifies the default storage root location for tables. When not specified, it defaults to the current working directory. ## Object Mapping ### Namespace -The **root namespace** is represented by the Iceberg catalog root, accessed via the `/namespaces` endpoint. +The **root namespace** (empty identifier) represents the Iceberg REST Catalog server itself. -A **child namespace** is a nested namespace in Iceberg. Iceberg supports arbitrary nesting depth using an array of strings (e.g., `["level1", "level2", "level3"]`). +The **warehouse** is the first level of the namespace hierarchy. The implementation caches the warehouse to config mapping by calling the `/v1/config?warehouse={warehouse}` endpoint. If the config response contains a `prefix` in the defaults, that prefix is used for API paths; otherwise, the warehouse name itself is used as the prefix. A single-element identifier (e.g., `["my-warehouse"]`) lists all top-level namespaces under that warehouse. -The **namespace identifier** is constructed by joining namespace levels with the `\x1F` (unit separator) character for API calls. In user-facing contexts, a `.` delimiter is used. +A **child namespace** is a nested namespace in Iceberg. Iceberg supports arbitrary nesting depth. The **namespace identifier** format is `[warehouse, namespace1, namespace2, ...]` (e.g., `["my-warehouse", "level1", "level2"]`). + +For API calls, namespace levels (excluding the warehouse) are joined with the `\x1F` (unit separator) character. In user-facing contexts, a `.` delimiter is used. **Namespace properties** are stored in the namespace's properties map, returned by the Iceberg namespace API. @@ -37,7 +42,7 @@ The **namespace identifier** is constructed by joining namespace levels with the A **table** is represented as a regular Iceberg table with a dummy schema. The dummy schema contains a single nullable string column named `dummy`. This approach ensures compatibility with the Iceberg REST Catalog API while storing the actual Lance table at the same location. -The **table identifier** is constructed by joining the namespace path and table name. +The **table identifier** format is `[warehouse, namespace1, namespace2, ..., table_name]` (e.g., `["my-warehouse", "db", "my_table"]`). The **table location** is stored in the `location` field of the Iceberg table metadata, pointing to the root location of the Lance table. @@ -55,14 +60,20 @@ Creates a new namespace in the Iceberg catalog. The implementation: -1. Parse the namespace identifier to get the namespace array -2. Construct a CreateNamespaceRequest with the namespace array and properties -3. POST to `/v1/{prefix}/namespaces` endpoint -4. Return the created namespace properties +1. Extract the warehouse from the first element of the namespace identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the remaining elements +4. Construct a CreateNamespaceRequest with the namespace array and properties +5. POST to `/v1/{prefix}/namespaces` endpoint +6. Return the created namespace properties **Error Handling:** -If the namespace already exists, return error code `2` (NamespaceAlreadyExists). If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace already exists, return error code `2` (NamespaceAlreadyExists). + +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### ListNamespaces @@ -70,13 +81,17 @@ Lists child namespaces under a given parent namespace. The implementation: -1. Parse the parent namespace identifier -2. GET `/v1/{prefix}/namespaces` with `parent` query parameter -3. Extract namespace names from the response +1. Extract the warehouse from the first element of the namespace identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the parent namespace path from the remaining elements (if any) +4. GET `/v1/{prefix}/namespaces` with `parent` query parameter +5. Extract namespace names from the response **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeNamespace @@ -84,13 +99,17 @@ Retrieves properties and metadata for a namespace. The implementation: -1. Parse the namespace identifier -2. GET `/v1/{prefix}/namespaces/{namespace}` with URL-encoded namespace path -3. Return the namespace properties +1. Extract the warehouse from the first element of the namespace identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the remaining elements +4. GET `/v1/{prefix}/namespaces/{namespace}` with URL-encoded namespace path +5. Return the namespace properties **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DropNamespace @@ -98,8 +117,10 @@ Removes a namespace from the Iceberg catalog. Only RESTRICT mode is supported; C The implementation: -1. Parse the namespace identifier -2. DELETE `/v1/{prefix}/namespaces/{namespace}` with URL-encoded namespace path +1. Extract the warehouse from the first element of the namespace identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the remaining elements +4. DELETE `/v1/{prefix}/namespaces/{namespace}` with URL-encoded namespace path **Error Handling:** @@ -115,18 +136,25 @@ Declares a new Lance table in the Iceberg catalog without creating the underlyin The implementation: -1. Parse the table identifier to extract namespace and table name -2. Construct a CreateTableRequest with: +1. Extract the warehouse from the first element of the table identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the middle elements +4. Extract the table name from the last element +5. Construct a CreateTableRequest with: - `name`: the table name - - `location`: the specified or default location (defaults to `{root}/{prefix}/{namespace}/{table_name}`) + - `location`: the specified or default location (defaults to `{root}/{warehouse}/{namespace}/{table_name}`) - `schema`: a dummy Iceberg schema with a single nullable string column `dummy` - `properties`: table properties including `table_type=lance` -3. POST to `/v1/{prefix}/namespaces/{namespace}/tables` -4. Return the declared table location +6. POST to `/v1/{prefix}/namespaces/{namespace}/tables` +7. Return the declared table location **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the table already exists, return error code `5` (TableAlreadyExists). If the server returns an error, return error code `18` (Internal). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the table already exists, return error code `5` (TableAlreadyExists). + +If the server returns an error, return error code `18` (Internal). ### ListTables @@ -134,14 +162,18 @@ Lists all Lance tables in a namespace. The implementation: -1. Parse the namespace identifier -2. GET `/v1/{prefix}/namespaces/{namespace}/tables` -3. For each table, load its metadata and filter tables where `properties.table_type=lance` -4. Extract table names from the response identifiers +1. Extract the warehouse from the first element of the namespace identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the remaining elements +4. GET `/v1/{prefix}/namespaces/{namespace}/tables` +5. For each table, load its metadata and filter tables where `properties.table_type=lance` +6. Extract table names from the response identifiers **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeTable @@ -149,10 +181,13 @@ Retrieves metadata for a Lance table. Only `load_detailed_metadata=false` is sup The implementation: -1. Parse the table identifier to extract namespace and table name -2. GET `/v1/{prefix}/namespaces/{namespace}/tables/{table}` -3. Verify the table has `table_type=lance` property -4. Return the table location and storage_options from `properties` +1. Extract the warehouse from the first element of the table identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the middle elements +4. Extract the table name from the last element +5. GET `/v1/{prefix}/namespaces/{namespace}/tables/{table}` +6. Verify the table has `table_type=lance` property +7. Return the table location and storage_options from `properties` **Error Handling:** @@ -168,9 +203,14 @@ Removes a Lance table registration from the Iceberg catalog without deleting the The implementation: -1. Parse the table identifier to extract namespace and table name -2. DELETE `/v1/{prefix}/namespaces/{namespace}/tables/{table}?purgeRequested=false` +1. Extract the warehouse from the first element of the table identifier +2. Resolve the API prefix from the warehouse config cache +3. Extract the namespace path from the middle elements +4. Extract the table name from the last element +5. DELETE `/v1/{prefix}/namespaces/{namespace}/tables/{table}?purgeRequested=false` **Error Handling:** -If the table does not exist, return error code `4` (TableNotFound). If the server returns an error, return error code `18` (Internal). +If the table does not exist, return error code `4` (TableNotFound). + +If the server returns an error, return error code `18` (Internal). diff --git a/docs/src/onelake.md b/docs/src/onelake.md index 068d8fe..571368b 100644 --- a/docs/src/onelake.md +++ b/docs/src/onelake.md @@ -1,4 +1,4 @@ -# Lance OneLake Namespace +# Azure OneLake Table API **Microsoft OneLake** is a unified, logical data lake for Microsoft Fabric that provides a single SaaS experience and a tenant-wide store for data that serves both professional and citizen data integration needs. diff --git a/docs/src/polaris.md b/docs/src/polaris.md index 77cd883..9c7bd3f 100644 --- a/docs/src/polaris.md +++ b/docs/src/polaris.md @@ -1,12 +1,12 @@ -# Lance Polaris Namespace Implementation Spec +# Apache Polaris Lance Namespace Implementation Spec This document describes how the Polaris Catalog implements the Lance Namespace client spec. ## Background -Apache Polaris is an open-source catalog implementation for Apache Iceberg that provides a REST API for managing tables and namespaces. Polaris supports the Generic Table API which allows registering non-Iceberg table formats. For details on Polaris Catalog, see the [Polaris Catalog Documentation](https://polaris.apache.org). - -**Note:** The Generic Table API is available in Polaris 1.2.0-incubating and later versions. Ensure your Polaris deployment is running a compatible version. +Apache Polaris is an open-source catalog implementation for Apache Iceberg that provides a REST API for managing tables and namespaces. +Polaris supports the Generic Table API which allows registering non-Iceberg table formats. +For details on Polaris Catalog, see the [Polaris Catalog Documentation](https://polaris.apache.org). ## Namespace Implementation Configuration Properties @@ -26,14 +26,13 @@ The **max_retries** property is optional and specifies the maximum number of ret ### Namespace -The **namespace identifier** follows a hierarchical structure where the first level represents the Polaris catalog (warehouse), and subsequent levels represent namespaces within that catalog. For example, `my_catalog.my_schema` refers to namespace `my_schema` in catalog `my_catalog`. +The **root namespace** (empty identifier) represents the Polaris server itself. + +The **catalog** is the first level of the namespace hierarchy. It determines which Polaris catalog the operations target. A single-element identifier (e.g., `["my-catalog"]`) lists all top-level namespaces under that catalog. -A **child namespace** is a nested namespace in Polaris. Polaris supports arbitrary nesting depth, allowing flexible namespace organization within a catalog. +A **child namespace** is a nested namespace in Polaris. Polaris supports arbitrary nesting depth. The **namespace identifier** format is `[catalog, namespace1, namespace2, ...]` (e.g., `["my-catalog", "schema", "subschema"]`). -The **namespace identifier** is constructed by joining the catalog and namespace levels with the `.` delimiter (e.g., `catalog.schema.subschema`). When making API calls: -- The catalog is extracted as the first level -- Remaining levels form the namespace path within that catalog -- The namespace path is URL-encoded using `.` as the separator +For API calls, namespace levels (excluding the catalog) are joined with the `.` delimiter. The catalog is used in the API path as `/api/catalog/v1/{catalog}/namespaces`. **Namespace properties** are stored in the namespace's properties map, returned by the Polaris namespace API. @@ -41,7 +40,7 @@ The **namespace identifier** is constructed by joining the catalog and namespace A **table** is represented as a [Generic Table](https://github.com/polaris-catalog/polaris/blob/main/spec/polaris-catalog-apis/generic-tables-api.yaml) object in Polaris with `format` set to `lance`. -The **table identifier** is constructed by joining the namespace path and table name with the `.` delimiter (e.g., `catalog.schema.table`). +The **table identifier** format is `[catalog, namespace1, namespace2, ..., table_name]` (e.g., `["my-catalog", "schema", "my_table"]`). The **table location** is stored in the `base-location` field of the Generic Table, pointing to the root location of the Lance table. @@ -67,7 +66,11 @@ The implementation: **Error Handling:** -If the namespace already exists, return error code `2` (NamespaceAlreadyExists). If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace already exists, return error code `2` (NamespaceAlreadyExists). + +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### ListNamespaces @@ -83,7 +86,9 @@ The implementation: **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeNamespace @@ -98,7 +103,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DropNamespace @@ -137,7 +144,11 @@ The implementation: **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the table already exists, return error code `5` (TableAlreadyExists). If the server returns an error, return error code `18` (Internal). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the table already exists, return error code `5` (TableAlreadyExists). + +If the server returns an error, return error code `18` (Internal). ### ListTables @@ -152,7 +163,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeTable @@ -186,4 +199,6 @@ The implementation: **Error Handling:** -If the table does not exist, return error code `4` (TableNotFound). If the server returns an error, return error code `18` (Internal). +If the table does not exist, return error code `4` (TableNotFound). + +If the server returns an error, return error code `18` (Internal). diff --git a/docs/src/unity.md b/docs/src/unity.md index 161038d..a8199dd 100644 --- a/docs/src/unity.md +++ b/docs/src/unity.md @@ -1,4 +1,4 @@ -# Lance Unity Namespace Implementation Spec +# Unity Catalog Lance Namespace Implementation Spec This document describes how the Unity Catalog implements the Lance Namespace client spec. @@ -26,11 +26,11 @@ The **root** property is optional and specifies the storage root location for ne ### Namespace -The **namespace identifier** follows a hierarchical structure where the first level represents the Unity Catalog, and the second level represents a schema within that catalog. For example, `my_catalog.my_schema` refers to schema `my_schema` in catalog `my_catalog`. +The **root namespace** (empty identifier) represents the Unity Catalog server itself. -The **root namespace** (empty identifier) lists all available catalogs in the Unity Catalog server. +The **catalog** is the first level of the namespace hierarchy. A single-element identifier (e.g., `["my-catalog"]`) lists all schemas within that catalog. -A **child namespace** is a schema within a Unity Catalog. Unity supports a fixed 3-level hierarchy: catalog.schema.table. +A **schema** is the second level of the namespace hierarchy. Unity supports a fixed 3-level hierarchy: catalog.schema.table. The **namespace identifier** format is `[catalog, schema]` (e.g., `["my-catalog", "my_schema"]`). **Namespace properties** are stored in the Unity schema's properties map. @@ -65,7 +65,9 @@ The implementation: **Error Handling:** -If the schema already exists, return error code `2` (NamespaceAlreadyExists). If the server returns an error, return error code `18` (Internal). +If the schema already exists, return error code `2` (NamespaceAlreadyExists). + +If the server returns an error, return error code `18` (Internal). ### ListNamespaces @@ -80,7 +82,9 @@ The implementation: **Error Handling:** -If the catalog does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the catalog does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeNamespace @@ -94,7 +98,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DropNamespace @@ -133,7 +139,11 @@ The implementation: **Error Handling:** -If the parent namespace does not exist, return error code `1` (NamespaceNotFound). If the table already exists, return error code `5` (TableAlreadyExists). If the server returns an error, return error code `18` (Internal). +If the parent namespace does not exist, return error code `1` (NamespaceNotFound). + +If the table already exists, return error code `5` (TableAlreadyExists). + +If the server returns an error, return error code `18` (Internal). ### ListTables @@ -148,7 +158,9 @@ The implementation: **Error Handling:** -If the namespace does not exist, return error code `1` (NamespaceNotFound). If the server returns an error, return error code `18` (Internal). +If the namespace does not exist, return error code `1` (NamespaceNotFound). + +If the server returns an error, return error code `18` (Internal). ### DescribeTable @@ -181,4 +193,8 @@ The implementation: **Error Handling:** -If the table does not exist, return error code `4` (TableNotFound). If the table is not a Lance table, return error code `13` (InvalidInput). If the server returns an error, return error code `18` (Internal). +If the table does not exist, return error code `4` (TableNotFound). + +If the table is not a Lance table, return error code `13` (InvalidInput). + +If the server returns an error, return error code `18` (Internal). diff --git a/java/Makefile b/java/Makefile index e1eea6e..ac70fc4 100644 --- a/java/Makefile +++ b/java/Makefile @@ -42,7 +42,7 @@ build-hive2: .PHONY: test-hive2 test-hive2: - ./mvnw test -pl lance-namespace-hive2 -Dtest="!*IntegrationTest" + ./mvnw test -pl lance-namespace-hive2 -Dtest="!*Integration" # ============================================================================ # Hive3 @@ -59,7 +59,7 @@ build-hive3: .PHONY: test-hive3 test-hive3: - ./mvnw test -pl lance-namespace-hive3 -Dtest="!*IntegrationTest" + ./mvnw test -pl lance-namespace-hive3 -Dtest="!*Integration" # ============================================================================ # Iceberg @@ -76,7 +76,7 @@ build-iceberg: .PHONY: test-iceberg test-iceberg: - ./mvnw test -pl lance-namespace-iceberg -Dtest="!*IntegrationTest" + ./mvnw test -pl lance-namespace-iceberg -Dtest="!*Integration" # ============================================================================ # Polaris @@ -93,7 +93,7 @@ build-polaris: .PHONY: test-polaris test-polaris: - ./mvnw test -pl lance-namespace-polaris -Dtest="!*IntegrationTest" + ./mvnw test -pl lance-namespace-polaris -Dtest="!*Integration" # ============================================================================ # Unity @@ -110,7 +110,7 @@ build-unity: .PHONY: test-unity test-unity: - ./mvnw test -pl lance-namespace-unity -Dtest="!*IntegrationTest" + ./mvnw test -pl lance-namespace-unity -Dtest="!*Integration" # ============================================================================ # All diff --git a/java/lance-namespace-glue/pom.xml b/java/lance-namespace-glue/pom.xml index cb8576c..392c7b6 100644 --- a/java/lance-namespace-glue/pom.xml +++ b/java/lance-namespace-glue/pom.xml @@ -8,7 +8,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespace.java b/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespace.java index 847c691..82af17e 100644 --- a/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespace.java +++ b/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespace.java @@ -19,10 +19,10 @@ import org.lance.namespace.errors.InvalidInputException; import org.lance.namespace.errors.NamespaceNotFoundException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -286,10 +286,8 @@ public DeregisterTableResponse deregisterTable(DeregisterTableRequest request) { } } - // Removed: createTable(CreateTableRequest, byte[]) - using default implementation from interface - @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { + public DeclareTableResponse declareTable(DeclareTableRequest request) { validateTableId(request.getId()); String namespaceName = request.getId().get(0); String tableName = request.getId().get(1); @@ -319,7 +317,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request .tableInput(tableInput) .build()); - CreateEmptyTableResponse response = new CreateEmptyTableResponse(); + DeclareTableResponse response = new DeclareTableResponse(); response.setLocation(location); response.setStorageOptions(config.getStorageOptions()); return response; @@ -331,7 +329,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request e, "Namespace not found: %s", namespaceName); } catch (GlueException e) { throw GlueToLanceErrorConverter.serverError( - e, "Failed to create empty table: %s.%s", namespaceName, tableName); + e, "Failed to declare table: %s.%s", namespaceName, tableName); } } diff --git a/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespaceConfig.java b/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespaceConfig.java index 0defed3..9c2ce8a 100644 --- a/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespaceConfig.java +++ b/java/lance-namespace-glue/src/main/java/org/lance/namespace/glue/GlueNamespaceConfig.java @@ -47,7 +47,11 @@ public class GlueNamespaceConfig implements Serializable { */ public static final String ENDPOINT = "endpoint"; - /** Configure the AWS region for all Glue operations */ + /** + * Configure the AWS region for all Glue operations. + * + *

When not specified, it is resolved to the default AWS region in the caller's environment. + */ public static final String REGION = "region"; /** diff --git a/java/lance-namespace-glue/src/test/java/org/lance/namespace/glue/TestGlueNamespaceIntegration.java b/java/lance-namespace-glue/src/test/java/org/lance/namespace/glue/TestGlueNamespaceIntegration.java index e37d6bf..d1e456a 100644 --- a/java/lance-namespace-glue/src/test/java/org/lance/namespace/glue/TestGlueNamespaceIntegration.java +++ b/java/lance-namespace-glue/src/test/java/org/lance/namespace/glue/TestGlueNamespaceIntegration.java @@ -13,11 +13,10 @@ */ package org.lance.namespace.glue; -import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -28,8 +27,6 @@ import org.lance.namespace.model.ListNamespacesResponse; import org.lance.namespace.model.ListTablesRequest; import org.lance.namespace.model.ListTablesResponse; -import org.lance.namespace.model.NamespaceExistsRequest; -import org.lance.namespace.model.TableExistsRequest; import org.apache.arrow.memory.BufferAllocator; import org.apache.arrow.memory.RootAllocator; @@ -48,7 +45,6 @@ import java.util.UUID; import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.assertThatThrownBy; /** * Integration tests for GlueNamespace against a real AWS Glue catalog. @@ -141,11 +137,13 @@ public void setUp() { @AfterEach public void tearDown() { // Clean up test resources - for (String dbName : createdDatabases) { - try { - cleanupDatabase(dbName); - } catch (Exception e) { - // Ignore cleanup errors + if (createdDatabases != null) { + for (String dbName : createdDatabases) { + try { + cleanupDatabase(dbName); + } catch (Exception e) { + // Ignore cleanup errors + } } } @@ -219,11 +217,6 @@ public void testNamespaceOperations() { assertThat(describeResponse.getProperties()) .containsEntry("description", "Test database for Lance"); - // Check namespace exists - NamespaceExistsRequest existsRequest = new NamespaceExistsRequest(); - existsRequest.setId(Collections.singletonList(dbName)); - namespace.namespaceExists(existsRequest); // Should not throw - // List namespaces ListNamespacesRequest listRequest = new ListNamespacesRequest(); listRequest.setId(Collections.emptyList()); @@ -235,10 +228,6 @@ public void testNamespaceOperations() { dropRequest.setId(Collections.singletonList(dbName)); namespace.dropNamespace(dropRequest); createdDatabases.remove(dbName); - - // Verify namespace doesn't exist - assertThatThrownBy(() -> namespace.namespaceExists(existsRequest)) - .isInstanceOf(LanceNamespaceException.class); } @Test @@ -247,12 +236,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8); String tableLocation = s3Root + "/" + dbName + "/" + tableName + ".lance"; - // Create empty table - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(dbName, tableName)); createRequest.setLocation(tableLocation); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); assertThat(createResponse.getLocation()).isEqualTo(tableLocation); @@ -264,11 +253,6 @@ public void testTableOperations() { assertThat(describeResponse.getLocation()).isNotNull(); assertThat(describeResponse.getLocation()).isEqualTo(tableLocation); - // Check table exists - TableExistsRequest existsRequest = new TableExistsRequest(); - existsRequest.setId(Arrays.asList(dbName, tableName)); - namespace.tableExists(existsRequest); // Should not throw - // List tables ListTablesRequest listRequest = new ListTablesRequest(); listRequest.setId(Collections.singletonList(dbName)); @@ -280,10 +264,6 @@ public void testTableOperations() { DeregisterTableRequest deregisterRequest = new DeregisterTableRequest(); deregisterRequest.setId(Arrays.asList(dbName, tableName)); namespace.deregisterTable(deregisterRequest); - - // Verify table doesn't exist - assertThatThrownBy(() -> namespace.tableExists(existsRequest)) - .isInstanceOf(LanceNamespaceException.class); } @Test @@ -297,10 +277,10 @@ public void testMultipleTablesInNamespace() { tableNames.add(tableName); String tableLocation = s3Root + "/" + dbName + "/" + tableName + ".lance"; - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(dbName, tableName)); createRequest.setLocation(tableLocation); - namespace.createEmptyTable(createRequest); + namespace.declareTable(createRequest); } // List tables and verify all are present diff --git a/java/lance-namespace-hive2/pom.xml b/java/lance-namespace-hive2/pom.xml index 52585f1..7a9b4f0 100644 --- a/java/lance-namespace-hive2/pom.xml +++ b/java/lance-namespace-hive2/pom.xml @@ -8,7 +8,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2Namespace.java b/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2Namespace.java index 7c07ac2..24f2fef 100644 --- a/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2Namespace.java +++ b/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2Namespace.java @@ -22,10 +22,10 @@ import org.lance.namespace.errors.ServiceUnavailableException; import org.lance.namespace.errors.TableAlreadyExistsException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -287,14 +287,13 @@ public DescribeTableResponse describeTable(DescribeTableRequest request) { DescribeTableResponse response = new DescribeTableResponse(); response.setLocation(location.get()); - response.setStorageOptions(config.getStorageOptions()); return response; } // Removed: createTable(CreateTableRequest, byte[]) - using default implementation from interface @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { + public DeclareTableResponse declareTable(DeclareTableRequest request) { ObjectIdentifier tableId = ObjectIdentifier.of(request.getId()); ValidationUtil.checkArgument( @@ -308,9 +307,8 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request // Create table in metastore without data (pass null for requestData) doCreateTable(tableId, null, location, null, null); - CreateEmptyTableResponse response = new CreateEmptyTableResponse(); + DeclareTableResponse response = new DeclareTableResponse(); response.setLocation(location); - response.setStorageOptions(config.getStorageOptions()); return response; } @@ -476,10 +474,7 @@ protected void doCreateTable( if (data != null && data.length > 0) { WriteParams writeParams = - new WriteParams.Builder() - .withMode(WriteParams.WriteMode.CREATE) - .withStorageOptions(config.getStorageOptions()) - .build(); + new WriteParams.Builder().withMode(WriteParams.WriteMode.CREATE).build(); Dataset.create(allocator, location, schema, writeParams); } } catch (TException | InterruptedException | RuntimeException e) { @@ -637,15 +632,15 @@ private String getDefaultTableLocation(String namespaceName, String tableName) { if (!dbLocation.endsWith("/")) { dbLocation += "/"; } - return dbLocation + tableName.toLowerCase() + ".lance"; + return dbLocation + tableName.toLowerCase(); } } catch (Exception e) { // Fall back to using root config if database location fails LOG.warn("Failed to get database location for {}, using root config", namespaceName, e); } - // Use the configured root as fallback + // Use the configured root as fallback: {root}/{database}.db/{table} return String.format( - "%s/%s/%s.lance", config.getRoot(), namespaceName.toLowerCase(), tableName.toLowerCase()); + "%s/%s.db/%s", config.getRoot(), namespaceName.toLowerCase(), tableName.toLowerCase()); } } diff --git a/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2NamespaceConfig.java b/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2NamespaceConfig.java index 28a5e41..8ea3270 100644 --- a/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2NamespaceConfig.java +++ b/java/lance-namespace-hive2/src/main/java/org/lance/namespace/hive2/Hive2NamespaceConfig.java @@ -13,7 +13,6 @@ */ package org.lance.namespace.hive2; -import java.util.HashMap; import java.util.Map; public class Hive2NamespaceConfig { @@ -27,15 +26,12 @@ public class Hive2NamespaceConfig { public static final String CLIENT_POOL_SIZE = "client.pool-size"; public static final int CLIENT_POOL_SIZE_DEFAULT = 3; - public static final String STORAGE_OPTIONS_PREFIX = "storage."; - /** Storage root location of the lakehouse on Hive catalog */ public static final String ROOT = "root"; public static final String ROOT_DEFAULT = System.getProperty("user.dir"); private final int clientPoolSize; - private final Map storageOptions; private final String root; public Hive2NamespaceConfig(Map properties) { @@ -44,16 +40,6 @@ public Hive2NamespaceConfig(Map properties) { this.clientPoolSize = clientPoolSizeStr != null ? Integer.parseInt(clientPoolSizeStr) : CLIENT_POOL_SIZE_DEFAULT; - // Inline PropertyUtil.propertiesWithPrefix - Map filteredStorageOptions = new HashMap<>(); - for (Map.Entry entry : properties.entrySet()) { - if (entry.getKey().startsWith(STORAGE_OPTIONS_PREFIX)) { - filteredStorageOptions.put( - entry.getKey().substring(STORAGE_OPTIONS_PREFIX.length()), entry.getValue()); - } - } - this.storageOptions = filteredStorageOptions; - // Inline PropertyUtil.propertyAsString and OpenDalUtil.stripTrailingSlash String rootValue = properties.getOrDefault(ROOT, ROOT_DEFAULT); this.root = @@ -66,10 +52,6 @@ public int getClientPoolSize() { return clientPoolSize; } - public Map getStorageOptions() { - return storageOptions; - } - public String getRoot() { return root; } diff --git a/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceConfig.java b/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceConfig.java index 0b7a05c..2de2eb6 100644 --- a/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceConfig.java +++ b/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceConfig.java @@ -57,12 +57,10 @@ public void testOtherConfigurationValues() { Map properties = ImmutableMap.of( "root", "/custom/root", - "client.pool-size", "5", - "storage.s3.region", "us-west-2"); + "client.pool-size", "5"); Hive2NamespaceConfig config = new Hive2NamespaceConfig(properties); assertEquals("/custom/root", config.getRoot()); assertEquals(5, config.getClientPoolSize()); - assertEquals("us-west-2", config.getStorageOptions().get("s3.region")); } } diff --git a/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceIntegration.java b/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceIntegration.java index e6ba013..e6117d5 100644 --- a/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceIntegration.java +++ b/java/lance-namespace-hive2/src/test/java/org/lance/namespace/hive2/TestHive2NamespaceIntegration.java @@ -15,10 +15,10 @@ import org.lance.namespace.errors.InvalidInputException; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -188,12 +188,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8).replace("-", ""); - // Create empty table (declare table without data) - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(testDatabase, tableName)); createRequest.setLocation("/tmp/lance-integration-test/" + testDatabase + "/" + tableName); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); // Describe table diff --git a/java/lance-namespace-hive3/pom.xml b/java/lance-namespace-hive3/pom.xml index 54616be..7949e29 100644 --- a/java/lance-namespace-hive3/pom.xml +++ b/java/lance-namespace-hive3/pom.xml @@ -8,7 +8,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3Namespace.java b/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3Namespace.java index 258332a..66fd60a 100644 --- a/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3Namespace.java +++ b/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3Namespace.java @@ -23,10 +23,10 @@ import org.lance.namespace.errors.ServiceUnavailableException; import org.lance.namespace.errors.TableAlreadyExistsException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -308,7 +308,7 @@ public DescribeTableResponse describeTable(DescribeTableRequest request) { // Removed: createTable(CreateTableRequest, byte[]) - using default implementation from interface @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { + public DeclareTableResponse declareTable(DeclareTableRequest request) { ObjectIdentifier tableId = ObjectIdentifier.of(request.getId()); ValidationUtil.checkArgument( @@ -324,7 +324,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request // Create table in metastore without data (pass null for requestData and properties) doCreateTable(tableId, null, location, null, null); - CreateEmptyTableResponse response = new CreateEmptyTableResponse(); + DeclareTableResponse response = new DeclareTableResponse(); response.setLocation(location); return response; } @@ -534,10 +534,7 @@ protected void doCreateTable( if (data != null && data.length > 0) { WriteParams writeParams = - new WriteParams.Builder() - .withMode(WriteParams.WriteMode.CREATE) - .withStorageOptions(config.getStorageOptions()) - .build(); + new WriteParams.Builder().withMode(WriteParams.WriteMode.CREATE).build(); Dataset.create(allocator, location, schema, writeParams); } } @@ -743,7 +740,7 @@ private String getDefaultTableLocation(String catalogName, String dbName, String if (!dbLocation.endsWith("/")) { dbLocation += "/"; } - return dbLocation + tableName.toLowerCase() + ".lance"; + return dbLocation + tableName.toLowerCase(); } } catch (Exception e) { // Fall back to using root config if database location fails @@ -751,8 +748,14 @@ private String getDefaultTableLocation(String catalogName, String dbName, String "Failed to get database location for {}.{}, using root config", catalogName, dbName, e); } - // Use the configured root as fallback + // For default "hive" catalog, use hive2-compatible path: {root}/{database}.db/{table} + // For other catalogs, use: {root}/{catalog}/{database}.db/{table} + if ("hive".equalsIgnoreCase(catalogName)) { + return String.format( + "%s/%s.db/%s", config.getRoot(), dbName.toLowerCase(), tableName.toLowerCase()); + } return String.format( - "%s/%s/%s.lance", config.getRoot(), dbName.toLowerCase(), tableName.toLowerCase()); + "%s/%s/%s.db/%s", + config.getRoot(), catalogName.toLowerCase(), dbName.toLowerCase(), tableName.toLowerCase()); } } diff --git a/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3NamespaceConfig.java b/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3NamespaceConfig.java index c5ba2a2..f598930 100644 --- a/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3NamespaceConfig.java +++ b/java/lance-namespace-hive3/src/main/java/org/lance/namespace/hive3/Hive3NamespaceConfig.java @@ -13,7 +13,6 @@ */ package org.lance.namespace.hive3; -import java.util.HashMap; import java.util.Map; public class Hive3NamespaceConfig { @@ -28,15 +27,12 @@ public class Hive3NamespaceConfig { public static final String CLIENT_POOL_SIZE = "client.pool-size"; public static final int CLIENT_POOL_SIZE_DEFAULT = 3; - public static final String STORAGE_OPTIONS_PREFIX = "storage."; - /** Storage root location of the lakehouse on Hive catalog */ public static final String ROOT = "root"; public static final String ROOT_DEFAULT = System.getProperty("user.dir"); private final int clientPoolSize; - private final Map storageOptions; private final String root; public Hive3NamespaceConfig(Map properties) { @@ -45,16 +41,6 @@ public Hive3NamespaceConfig(Map properties) { this.clientPoolSize = clientPoolSizeStr != null ? Integer.parseInt(clientPoolSizeStr) : CLIENT_POOL_SIZE_DEFAULT; - // Inline PropertyUtil.propertiesWithPrefix - Map filteredStorageOptions = new HashMap<>(); - for (Map.Entry entry : properties.entrySet()) { - if (entry.getKey().startsWith(STORAGE_OPTIONS_PREFIX)) { - filteredStorageOptions.put( - entry.getKey().substring(STORAGE_OPTIONS_PREFIX.length()), entry.getValue()); - } - } - this.storageOptions = filteredStorageOptions; - // Inline PropertyUtil.propertyAsString and OpenDalUtil.stripTrailingSlash String rootValue = properties.getOrDefault(ROOT, ROOT_DEFAULT); this.root = @@ -67,10 +53,6 @@ public int getClientPoolSize() { return clientPoolSize; } - public Map getStorageOptions() { - return storageOptions; - } - public String getRoot() { return root; } diff --git a/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceConfig.java b/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceConfig.java index 4947491..9d6ef8b 100644 --- a/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceConfig.java +++ b/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceConfig.java @@ -57,12 +57,10 @@ public void testOtherConfigurationValues() { Map properties = ImmutableMap.of( "root", "/custom/root", - "client.pool-size", "5", - "storage.s3.region", "us-west-2"); + "client.pool-size", "5"); Hive3NamespaceConfig config = new Hive3NamespaceConfig(properties); assertEquals("/custom/root", config.getRoot()); assertEquals(5, config.getClientPoolSize()); - assertEquals("us-west-2", config.getStorageOptions().get("s3.region")); } } diff --git a/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceIntegration.java b/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceIntegration.java index 8e26396..b5a8eae 100644 --- a/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceIntegration.java +++ b/java/lance-namespace-hive3/src/test/java/org/lance/namespace/hive3/TestHive3NamespaceIntegration.java @@ -15,10 +15,10 @@ import org.lance.namespace.errors.InvalidInputException; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -188,12 +188,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8).replace("-", ""); - // Create empty table (declare table without data) - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(testCatalog, testDatabase, tableName)); createRequest.setLocation("/tmp/lance-integration-test/" + testDatabase + "/" + tableName); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); // Describe table diff --git a/java/lance-namespace-hive3/src/test/resources/integration-test.properties b/java/lance-namespace-hive3/src/test/resources/integration-test.properties index 2144629..9df832a 100644 --- a/java/lance-namespace-hive3/src/test/resources/integration-test.properties +++ b/java/lance-namespace-hive3/src/test/resources/integration-test.properties @@ -9,7 +9,3 @@ client.pool-size=3 # Storage root for test tables root=/tmp/lance-integration-test - -# Optional: storage options for cloud storage -# storage.region=us-west-2 -# storage.endpoint=http://localhost:9000 diff --git a/java/lance-namespace-iceberg/pom.xml b/java/lance-namespace-iceberg/pom.xml index bfa5d9d..351edfc 100644 --- a/java/lance-namespace-iceberg/pom.xml +++ b/java/lance-namespace-iceberg/pom.xml @@ -8,7 +8,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespace.java b/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespace.java index 9eb2050..ca7038c 100644 --- a/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespace.java +++ b/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespace.java @@ -20,8 +20,6 @@ import org.lance.namespace.errors.NamespaceNotFoundException; import org.lance.namespace.errors.TableAlreadyExistsException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; import org.lance.namespace.model.DeclareTableRequest; @@ -65,14 +63,16 @@ /** * Iceberg REST Catalog namespace implementation for Lance. * - *

The prefix (warehouse) is included in the namespace identifier: + *

The warehouse is the first element of the namespace/table identifier: * *

* - *

This is consistent with how Polaris handles catalog names. + *

The implementation caches warehouse -> config mappings by calling + * /v1/config?warehouse={warehouse}. If the config contains a prefix, that prefix is used for API + * paths; otherwise, the warehouse name is used. */ public class IcebergNamespace implements LanceNamespace, Closeable { private static final Logger LOG = LoggerFactory.getLogger(IcebergNamespace.class); @@ -102,9 +102,6 @@ public void initialize(Map configProperties, BufferAllocator all if (config.getAuthToken() != null) { clientBuilder.authToken(config.getAuthToken()); } - if (config.getWarehouse() != null) { - clientBuilder.header("X-Iceberg-Access-Delegation", "vended-credentials"); - } this.restClient = clientBuilder.build(); LOG.info("Initialized Iceberg namespace with endpoint: {}", config.getEndpoint()); @@ -153,8 +150,7 @@ public ListNamespacesResponse listNamespaces(ListNamespacesRequest request) { ? ObjectIdentifier.of(request.getId()) : ObjectIdentifier.of(Collections.emptyList()); - ValidationUtil.checkArgument( - nsId.levels() >= 1, "Must specify at least the prefix (warehouse)"); + ValidationUtil.checkArgument(nsId.levels() >= 1, "Must specify at least the warehouse"); try { String prefix = nsId.levelAtListPos(0); @@ -302,7 +298,8 @@ public DropNamespaceResponse dropNamespace(DropNamespaceRequest request) { @Override public ListTablesResponse listTables(ListTablesRequest request) { ObjectIdentifier nsId = ObjectIdentifier.of(request.getId()); - ValidationUtil.checkArgument(nsId.levels() >= 2, "Must specify at least prefix and namespace"); + ValidationUtil.checkArgument( + nsId.levels() >= 2, "Must specify at least warehouse and namespace"); try { String prefix = nsId.levelAtListPos(0); @@ -399,21 +396,6 @@ public DeclareTableResponse declareTable(DeclareTableRequest request) { } } - /** - * @deprecated Use {@link #declareTable(DeclareTableRequest)} instead. - */ - @Deprecated - @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { - DeclareTableRequest declareRequest = new DeclareTableRequest(); - declareRequest.setId(request.getId()); - declareRequest.setLocation(request.getLocation()); - DeclareTableResponse response = declareTable(declareRequest); - CreateEmptyTableResponse result = new CreateEmptyTableResponse(); - result.setLocation(response.getLocation()); - return result; - } - @Override public DescribeTableResponse describeTable(DescribeTableRequest request) { if (Boolean.TRUE.equals(request.getLoadDetailedMetadata())) { diff --git a/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespaceConfig.java b/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespaceConfig.java index ce1793c..2c74b60 100644 --- a/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespaceConfig.java +++ b/java/lance-namespace-iceberg/src/main/java/org/lance/namespace/iceberg/IcebergNamespaceConfig.java @@ -19,7 +19,6 @@ public class IcebergNamespaceConfig { public static final String ENDPOINT = "endpoint"; - public static final String WAREHOUSE = "warehouse"; public static final String AUTH_TOKEN = "auth_token"; public static final String CREDENTIAL = "credential"; public static final String CONNECT_TIMEOUT = "connect_timeout"; @@ -28,7 +27,6 @@ public class IcebergNamespaceConfig { public static final String ROOT = "root"; private final String endpoint; - private final String warehouse; private final String authToken; private final String credential; private final int connectTimeout; @@ -42,7 +40,6 @@ public IcebergNamespaceConfig(Map properties) { throw new IllegalArgumentException("Required property 'endpoint' is not set"); } - this.warehouse = properties.get(WAREHOUSE); this.authToken = properties.get(AUTH_TOKEN); this.credential = properties.get(CREDENTIAL); this.connectTimeout = Integer.parseInt(properties.getOrDefault(CONNECT_TIMEOUT, "10000")); @@ -55,10 +52,6 @@ public String getEndpoint() { return endpoint; } - public String getWarehouse() { - return warehouse; - } - public String getAuthToken() { return authToken; } diff --git a/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceConfig.java b/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceConfig.java new file mode 100644 index 0000000..3fd685f --- /dev/null +++ b/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceConfig.java @@ -0,0 +1,99 @@ +/* + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.lance.namespace.iceberg; + +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; + +public class TestIcebergNamespaceConfig { + + @Test + public void testConfigWithEndpoint() { + Map properties = new HashMap<>(); + properties.put("endpoint", "http://localhost:8181"); + + IcebergNamespaceConfig config = new IcebergNamespaceConfig(properties); + + assertEquals("http://localhost:8181", config.getEndpoint()); + assertEquals(System.getProperty("user.dir"), config.getRoot()); + assertNull(config.getAuthToken()); + assertNull(config.getCredential()); + assertEquals(10000, config.getConnectTimeout()); + assertEquals(30000, config.getReadTimeout()); + assertEquals(3, config.getMaxRetries()); + } + + @Test + public void testConfigWithAllProperties() { + Map properties = new HashMap<>(); + properties.put("endpoint", "https://iceberg.example.com"); + properties.put("auth_token", "test_token"); + properties.put("credential", "client_id:client_secret"); + properties.put("connect_timeout", "5000"); + properties.put("read_timeout", "60000"); + properties.put("max_retries", "5"); + properties.put("root", "/data/lance"); + + IcebergNamespaceConfig config = new IcebergNamespaceConfig(properties); + + assertEquals("https://iceberg.example.com", config.getEndpoint()); + assertEquals("test_token", config.getAuthToken()); + assertEquals("client_id:client_secret", config.getCredential()); + assertEquals(5000, config.getConnectTimeout()); + assertEquals(60000, config.getReadTimeout()); + assertEquals(5, config.getMaxRetries()); + assertEquals("/data/lance", config.getRoot()); + } + + @Test + public void testMissingEndpointThrowsException() { + Map properties = new HashMap<>(); + + assertThrows(IllegalArgumentException.class, () -> new IcebergNamespaceConfig(properties)); + } + + @Test + public void testEmptyEndpointThrowsException() { + Map properties = new HashMap<>(); + properties.put("endpoint", ""); + + assertThrows(IllegalArgumentException.class, () -> new IcebergNamespaceConfig(properties)); + } + + @Test + public void testGetBaseApiUrlStripsTrailingSlash() { + Map properties = new HashMap<>(); + properties.put("endpoint", "http://localhost:8181/"); + + IcebergNamespaceConfig config = new IcebergNamespaceConfig(properties); + + assertEquals("http://localhost:8181", config.getBaseApiUrl()); + } + + @Test + public void testGetBaseApiUrlWithoutTrailingSlash() { + Map properties = new HashMap<>(); + properties.put("endpoint", "http://localhost:8181"); + + IcebergNamespaceConfig config = new IcebergNamespaceConfig(properties); + + assertEquals("http://localhost:8181", config.getBaseApiUrl()); + } +} diff --git a/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceIntegration.java b/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceIntegration.java index ad7eb7c..4ed3745 100644 --- a/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceIntegration.java +++ b/java/lance-namespace-iceberg/src/test/java/org/lance/namespace/iceberg/TestIcebergNamespaceIntegration.java @@ -14,10 +14,10 @@ package org.lance.namespace.iceberg; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -191,12 +191,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8); - // Create empty table - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(TEST_WAREHOUSE, testNamespace, tableName)); createRequest.setLocation("s3://warehouse/" + testNamespace + "/" + tableName); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); // Describe table @@ -230,18 +230,18 @@ public void testTableOperations() { } @Test - public void testCreateEmptyTableWithLocation() { + public void testDeclareTableWithLocation() { // Create namespace first CreateNamespaceRequest nsRequest = new CreateNamespaceRequest(); nsRequest.setId(Arrays.asList(TEST_WAREHOUSE, testNamespace)); namespace.createNamespace(nsRequest); String tableName = "lance_table"; - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(TEST_WAREHOUSE, testNamespace, tableName)); createRequest.setLocation("s3://warehouse/" + testNamespace + "/" + tableName); - CreateEmptyTableResponse response = namespace.createEmptyTable(createRequest); + DeclareTableResponse response = namespace.declareTable(createRequest); assertThat(response.getLocation()).isNotNull(); // Clean up table diff --git a/java/lance-namespace-impls-core/pom.xml b/java/lance-namespace-impls-core/pom.xml index 878d8c3..874024b 100644 --- a/java/lance-namespace-impls-core/pom.xml +++ b/java/lance-namespace-impls-core/pom.xml @@ -7,7 +7,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 lance-namespace-impls-core diff --git a/java/lance-namespace-polaris/pom.xml b/java/lance-namespace-polaris/pom.xml index cf71b17..4d05466 100644 --- a/java/lance-namespace-polaris/pom.xml +++ b/java/lance-namespace-polaris/pom.xml @@ -20,7 +20,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-polaris/src/main/java/org/lance/namespace/polaris/PolarisNamespace.java b/java/lance-namespace-polaris/src/main/java/org/lance/namespace/polaris/PolarisNamespace.java index b70461a..21f63a3 100644 --- a/java/lance-namespace-polaris/src/main/java/org/lance/namespace/polaris/PolarisNamespace.java +++ b/java/lance-namespace-polaris/src/main/java/org/lance/namespace/polaris/PolarisNamespace.java @@ -20,10 +20,10 @@ import org.lance.namespace.errors.NamespaceNotFoundException; import org.lance.namespace.errors.TableAlreadyExistsException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -274,7 +274,7 @@ public void tableExists(TableExistsRequest request) { } @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { + public DeclareTableResponse declareTable(DeclareTableRequest request) { ObjectIdentifier tableId = ObjectIdentifier.of(request.getId()); ValidationUtil.checkArgument( tableId.levels() >= 3, "Table identifier must have catalog, namespace, and table name"); @@ -302,7 +302,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request LOG.info("Created Lance table: {}.{}.{}", catalog, namespacePath, tableName); - CreateEmptyTableResponse result = new CreateEmptyTableResponse(); + DeclareTableResponse result = new DeclareTableResponse(); result.setLocation(response.getTable().getBaseLocation()); return result; } catch (RestClientException e) { diff --git a/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespace.java b/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespace.java index 55d7622..71f4c30 100644 --- a/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespace.java +++ b/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespace.java @@ -14,10 +14,10 @@ package org.lance.namespace.polaris; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -210,7 +210,7 @@ public void testNamespaceNotExists() throws IOException { } @Test - public void testCreateEmptyTable() throws IOException { + public void testDeclareTable() throws IOException { PolarisModels.GenericTable mockTable = new PolarisModels.GenericTable(); mockTable.setName("test_table"); mockTable.setFormat("lance"); @@ -230,11 +230,11 @@ public void testCreateEmptyTable() throws IOException { eq(PolarisModels.LoadGenericTableResponse.class))) .thenReturn(mockResponse); - CreateEmptyTableRequest request = new CreateEmptyTableRequest(); + DeclareTableRequest request = new DeclareTableRequest(); request.setId(Arrays.asList("test_catalog", "schema1", "test_table")); request.setLocation("s3://bucket/path/to/table"); - CreateEmptyTableResponse response = namespace.createEmptyTable(request); + DeclareTableResponse response = namespace.declareTable(request); assertThat(response.getLocation()).isEqualTo("s3://bucket/path/to/table"); } diff --git a/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespaceIntegration.java b/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespaceIntegration.java index e7a4804..5fbcf00 100644 --- a/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespaceIntegration.java +++ b/java/lance-namespace-polaris/src/test/java/org/lance/namespace/polaris/TestPolarisNamespaceIntegration.java @@ -14,10 +14,10 @@ package org.lance.namespace.polaris; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -248,12 +248,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8); - // Create empty table - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(testCatalog, testNamespace, tableName)); createRequest.setLocation("/tmp/polaris-test/" + testNamespace + "/" + tableName); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); // Describe table @@ -287,7 +287,7 @@ public void testTableOperations() { } @Test - public void testCreateEmptyTableWithLocation() { + public void testDeclareTableWithLocation() { // Create namespace first CreateNamespaceRequest nsRequest = new CreateNamespaceRequest(); nsRequest.setId(Arrays.asList(testCatalog, testNamespace)); @@ -295,11 +295,11 @@ public void testCreateEmptyTableWithLocation() { // Create an empty Lance table with location String tableName = "lance_table"; - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(testCatalog, testNamespace, tableName)); createRequest.setLocation("/tmp/polaris-test/" + testNamespace + "/" + tableName); - CreateEmptyTableResponse response = namespace.createEmptyTable(createRequest); + DeclareTableResponse response = namespace.declareTable(createRequest); assertThat(response.getLocation()).isNotNull(); } } diff --git a/java/lance-namespace-unity/pom.xml b/java/lance-namespace-unity/pom.xml index 767707a..7cc67c3 100644 --- a/java/lance-namespace-unity/pom.xml +++ b/java/lance-namespace-unity/pom.xml @@ -8,7 +8,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 ../pom.xml diff --git a/java/lance-namespace-unity/src/main/java/org/lance/namespace/unity/UnityNamespace.java b/java/lance-namespace-unity/src/main/java/org/lance/namespace/unity/UnityNamespace.java index 6dc5d1e..faba9d9 100644 --- a/java/lance-namespace-unity/src/main/java/org/lance/namespace/unity/UnityNamespace.java +++ b/java/lance-namespace-unity/src/main/java/org/lance/namespace/unity/UnityNamespace.java @@ -20,10 +20,10 @@ import org.lance.namespace.errors.NamespaceNotFoundException; import org.lance.namespace.errors.TableAlreadyExistsException; import org.lance.namespace.errors.TableNotFoundException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DeregisterTableResponse; import org.lance.namespace.model.DescribeNamespaceRequest; @@ -306,7 +306,7 @@ public ListTablesResponse listTables(ListTablesRequest request) { } @Override - public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request) { + public DeclareTableResponse declareTable(DeclareTableRequest request) { ObjectIdentifier tableId = ObjectIdentifier.of(request.getId()); ValidationUtil.checkArgument( tableId.levels() == 3, "Expect a 3-level table identifier but get %s", tableId); @@ -317,7 +317,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request if (!catalog.equals(config.getCatalog())) { throw new InvalidInputException( - "Cannot create empty table in catalog. Expected: " + config.getCatalog()); + "Cannot declare table in catalog. Expected: " + config.getCatalog()); } try { @@ -352,7 +352,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request UnityModels.TableInfo tableInfo = restClient.post("/tables", createTable, UnityModels.TableInfo.class); - CreateEmptyTableResponse response = new CreateEmptyTableResponse(); + DeclareTableResponse response = new DeclareTableResponse(); response.setLocation(tablePath); return response; @@ -361,7 +361,7 @@ public CreateEmptyTableResponse createEmptyTable(CreateEmptyTableRequest request throw new TableAlreadyExistsException( "Table already exists: " + request.getId().toString()); } - throw new InternalException("Failed to create empty table: " + e.getMessage()); + throw new InternalException("Failed to declare table: " + e.getMessage()); } } diff --git a/java/lance-namespace-unity/src/test/java/org/lance/namespace/unity/TestUnityNamespaceIntegration.java b/java/lance-namespace-unity/src/test/java/org/lance/namespace/unity/TestUnityNamespaceIntegration.java index 21033ae..1bce6c6 100644 --- a/java/lance-namespace-unity/src/test/java/org/lance/namespace/unity/TestUnityNamespaceIntegration.java +++ b/java/lance-namespace-unity/src/test/java/org/lance/namespace/unity/TestUnityNamespaceIntegration.java @@ -15,10 +15,10 @@ import org.lance.namespace.errors.InvalidInputException; import org.lance.namespace.errors.LanceNamespaceException; -import org.lance.namespace.model.CreateEmptyTableRequest; -import org.lance.namespace.model.CreateEmptyTableResponse; import org.lance.namespace.model.CreateNamespaceRequest; import org.lance.namespace.model.CreateNamespaceResponse; +import org.lance.namespace.model.DeclareTableRequest; +import org.lance.namespace.model.DeclareTableResponse; import org.lance.namespace.model.DeregisterTableRequest; import org.lance.namespace.model.DescribeNamespaceRequest; import org.lance.namespace.model.DescribeNamespaceResponse; @@ -236,12 +236,12 @@ public void testTableOperations() { String tableName = "test_table_" + UUID.randomUUID().toString().substring(0, 8).replace("-", ""); - // Create empty table - CreateEmptyTableRequest createRequest = new CreateEmptyTableRequest(); + // Declare table + DeclareTableRequest createRequest = new DeclareTableRequest(); createRequest.setId(Arrays.asList(TEST_CATALOG, testSchema, tableName)); createRequest.setLocation("/tmp/lance-integration-test/" + testSchema + "/" + tableName); - CreateEmptyTableResponse createResponse = namespace.createEmptyTable(createRequest); + DeclareTableResponse createResponse = namespace.declareTable(createRequest); assertThat(createResponse.getLocation()).isNotNull(); // Describe table diff --git a/java/pom.xml b/java/pom.xml index 7e57514..a80c8e6 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -7,7 +7,7 @@ org.lance lance-namespace-impls-root - 0.0.1 + 0.1.0 pom ${project.artifactId} diff --git a/python/pyproject.toml b/python/pyproject.toml index 055d965..f744a8f 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "lance-namespace-impls" -version = "0.0.1" +version = "0.1.0" description = "Third-party catalog implementations for Lance Namespace" readme = "README.md" license = { text = "Apache-2.0" } diff --git a/python/src/lance_namespace_impls/glue.py b/python/src/lance_namespace_impls/glue.py index 5ec803e..21c518f 100644 --- a/python/src/lance_namespace_impls/glue.py +++ b/python/src/lance_namespace_impls/glue.py @@ -26,8 +26,8 @@ DropNamespaceResponse, ListTablesRequest, ListTablesResponse, - CreateEmptyTableRequest, - CreateEmptyTableResponse, + DeclareTableRequest, + DeclareTableResponse, DescribeTableRequest, DescribeTableResponse, DeregisterTableRequest, @@ -98,9 +98,11 @@ def __init__(self, **properties): """Initialize the Glue namespace. Args: - catalog_id: Glue catalog ID (AWS account ID) + catalog_id: Glue catalog ID to use as starting point. When not specified, + it is resolved to the caller's AWS account ID. endpoint: Optional custom Glue endpoint - region: AWS region for Glue + region: AWS region for Glue. When not specified, it is resolved to the + default AWS region in the caller's environment. access_key_id: AWS access key ID secret_access_key: AWS secret access key session_token: AWS session token @@ -378,10 +380,8 @@ def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse raise raise RuntimeError(f"Failed to describe table: {e}") - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only) in Glue catalog.""" + def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: + """Declare a table (metadata only) in Glue catalog.""" database_name, table_name = self._parse_table_identifier(request.id) # Determine table location @@ -426,10 +426,6 @@ def create_empty_table( }, } - # Add additional properties if specified - if request.properties: - table_input["Parameters"].update(request.properties) - try: self.glue.create_table(DatabaseName=database_name, TableInput=table_input) except Exception as e: @@ -437,9 +433,9 @@ def create_empty_table( raise RuntimeError( f"Table already exists: {database_name}.{table_name}" ) - raise RuntimeError(f"Failed to create empty table: {e}") + raise RuntimeError(f"Failed to declare table: {e}") - return CreateEmptyTableResponse(location=table_location) + return DeclareTableResponse(location=table_location) def deregister_table( self, request: DeregisterTableRequest diff --git a/python/src/lance_namespace_impls/hive2.py b/python/src/lance_namespace_impls/hive2.py index 68bd488..47c9a2d 100644 --- a/python/src/lance_namespace_impls/hive2.py +++ b/python/src/lance_namespace_impls/hive2.py @@ -27,7 +27,6 @@ root (str): Storage root location of the lakehouse on Hive catalog (default: current working directory) ugi (str): Optional User Group Information for authentication (format: "user:group1,group2") client.pool-size (int): Size of the HMS client connection pool (default: 3) - storage.* (str): Additional storage configurations to access table """ from typing import List, Optional @@ -75,8 +74,8 @@ DropNamespaceResponse, ListTablesRequest, ListTablesResponse, - CreateEmptyTableRequest, - CreateEmptyTableResponse, + DeclareTableRequest, + DeclareTableResponse, DescribeTableRequest, DescribeTableResponse, DeregisterTableRequest, @@ -144,7 +143,6 @@ def __init__(self, **properties): root: Storage root location of the lakehouse on Hive catalog (optional) ugi: User Group Information for authentication (optional, format: "user:group1,group2") client.pool-size: Size of the HMS client connection pool (optional, default: 3) - storage.*: Additional storage configurations to access table **properties: Additional configuration properties """ if not HIVE_AVAILABLE: @@ -157,10 +155,6 @@ def __init__(self, **properties): self.ugi = properties.get("ugi") self.root = properties.get("root", os.getcwd()) self.pool_size = int(properties.get("client.pool-size", "3")) - # Extract storage properties - self.storage_properties = { - k[8:]: v for k, v in properties.items() if k.startswith("storage.") - } # Store properties for pickling support self._properties = properties.copy() @@ -371,7 +365,7 @@ def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse: """Describe a table in the Hive Metastore. - Only load_detailed_metadata=false is supported. Returns location and storage_options only. + Only load_detailed_metadata=false is supported. Returns location only. """ if request.load_detailed_metadata: raise ValueError( @@ -396,9 +390,7 @@ def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse if not location: raise ValueError(f"Table {request.id} has no location") - return DescribeTableResponse( - location=location, storage_options=self.storage_properties - ) + return DescribeTableResponse(location=location) except Exception as e: if NoSuchObjectException and isinstance(e, NoSuchObjectException): raise ValueError(f"Table {request.id} does not exist") @@ -435,10 +427,8 @@ def deregister_table( logger.error(f"Failed to deregister table {request.id}: {e}") raise - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only) in Hive metastore.""" + def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: + """Declare a table (metadata only) in Hive metastore.""" try: database, table_name = self._normalize_identifier(request.id) @@ -477,9 +467,6 @@ def create_empty_table( "empty_table": "true", # Mark as empty table } - if hasattr(request, "properties") and request.properties: - parameters.update(request.properties) - hive_table = HiveTable( tableName=table_name, dbName=database, @@ -492,12 +479,12 @@ def create_empty_table( with self.client as client: client.create_table(hive_table) - return CreateEmptyTableResponse(location=location) + return DeclareTableResponse(location=location) except AlreadyExistsException: raise ValueError(f"Table {request.id} already exists") except Exception as e: - logger.error(f"Failed to create empty table {request.id}: {e}") + logger.error(f"Failed to declare table {request.id}: {e}") raise def __getstate__(self): diff --git a/python/src/lance_namespace_impls/hive3.py b/python/src/lance_namespace_impls/hive3.py index 1b44f7f..32c9d6f 100644 --- a/python/src/lance_namespace_impls/hive3.py +++ b/python/src/lance_namespace_impls/hive3.py @@ -29,7 +29,6 @@ root (str): Storage root location of the lakehouse (default: current working directory) ugi (str): Optional User Group Information for authentication (format: "user:group1,group2") client.pool-size (int): Size of the HMS client connection pool (default: 3) - storage.* (str): Additional storage configurations """ from typing import List, Optional @@ -77,8 +76,8 @@ DropNamespaceResponse, ListTablesRequest, ListTablesResponse, - CreateEmptyTableRequest, - CreateEmptyTableResponse, + DeclareTableRequest, + DeclareTableResponse, DescribeTableRequest, DescribeTableResponse, DeregisterTableRequest, @@ -149,7 +148,6 @@ def __init__(self, **properties): root: Storage root location (optional) ugi: User Group Information for authentication (optional) client.pool-size: Size of the HMS client connection pool (optional, default: 3) - storage.*: Additional storage configurations **properties: Additional configuration properties """ if not HIVE_AVAILABLE: @@ -162,9 +160,6 @@ def __init__(self, **properties): self.ugi = properties.get("ugi") self.root = properties.get("root", os.getcwd()) self.pool_size = int(properties.get("client.pool-size", "3")) - self.storage_properties = { - k[8:]: v for k, v in properties.items() if k.startswith("storage.") - } self._properties = properties.copy() self._client = None @@ -197,7 +192,10 @@ def _is_root_namespace(self, identifier: Optional[List[str]]) -> bool: def _get_table_location(self, catalog: str, database: str, table: str) -> str: """Get the location for a table.""" - return os.path.join(self.root, database, f"{table}.lance") + if catalog.lower() == "hive": + # For default catalog, use hive2-compatible path + return os.path.join(self.root, f"{database}.db", table) + return os.path.join(self.root, catalog, f"{database}.db", table) def list_namespaces(self, request: ListNamespacesRequest) -> ListNamespacesResponse: """List namespaces at the given level. @@ -333,10 +331,10 @@ def create_namespace( ) database.locationUri = ( request.properties.get( - "location", os.path.join(self.root, database_name) + "location", os.path.join(self.root, f"{database_name}.db") ) if request.properties - else os.path.join(self.root, database_name) + else os.path.join(self.root, f"{database_name}.db") ) if request.properties: @@ -448,7 +446,7 @@ def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse: """Describe a table. - Only load_detailed_metadata=false is supported. Returns location and storage_options only. + Only load_detailed_metadata=false is supported. Returns location only. """ if request.load_detailed_metadata: raise ValueError( @@ -471,9 +469,7 @@ def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse if not location: raise ValueError(f"Table {request.id} has no location") - return DescribeTableResponse( - location=location, storage_options=self.storage_properties - ) + return DescribeTableResponse(location=location) except Exception as e: if NoSuchObjectException and isinstance(e, NoSuchObjectException): @@ -509,10 +505,8 @@ def deregister_table( logger.error(f"Failed to deregister table {request.id}: {e}") raise - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only).""" + def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: + """Declare a table (metadata only).""" try: catalog, database, table_name = self._normalize_identifier(request.id) @@ -545,9 +539,6 @@ def create_empty_table( "empty_table": "true", } - if hasattr(request, "properties") and request.properties: - parameters.update(request.properties) - hive_table = HiveTable( tableName=table_name, dbName=database, @@ -559,12 +550,12 @@ def create_empty_table( with self.client as client: client.create_table(hive_table) - return CreateEmptyTableResponse(location=location) + return DeclareTableResponse(location=location) except AlreadyExistsException: raise ValueError(f"Table {request.id} already exists") except Exception as e: - logger.error(f"Failed to create empty table {request.id}: {e}") + logger.error(f"Failed to declare table {request.id}: {e}") raise def __getstate__(self): diff --git a/python/src/lance_namespace_impls/iceberg.py b/python/src/lance_namespace_impls/iceberg.py index c655eee..89eaf12 100644 --- a/python/src/lance_namespace_impls/iceberg.py +++ b/python/src/lance_namespace_impls/iceberg.py @@ -1,21 +1,21 @@ """ Iceberg REST Catalog namespace implementation for Lance. -The prefix (typically a warehouse or catalog name) is treated as part of the -namespace identifier, similar to how Polaris handles catalog names. -For example: [warehouse_name, namespace1, namespace2, ..., table_name] +The warehouse is the first element of the namespace/table identifier. +For example: [warehouse, namespace1, namespace2, ..., table_name] + +The implementation caches warehouse -> config mappings by calling +/v1/config?warehouse={warehouse}. If the config contains a prefix, +that prefix is used for API paths; otherwise, the warehouse name is used. """ import logging import urllib.parse -import warnings from dataclasses import dataclass from typing import Any, Dict, List, Optional from lance.namespace import LanceNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, - CreateEmptyTableResponse, CreateNamespaceRequest, CreateNamespaceResponse, DeclareTableRequest, @@ -55,7 +55,6 @@ class IcebergNamespaceConfig: """Configuration for Iceberg REST Catalog namespace.""" ENDPOINT = "endpoint" - WAREHOUSE = "warehouse" AUTH_TOKEN = "auth_token" CREDENTIAL = "credential" CONNECT_TIMEOUT = "connect_timeout" @@ -64,7 +63,6 @@ class IcebergNamespaceConfig: ROOT = "root" endpoint: str - warehouse: Optional[str] = None auth_token: Optional[str] = None credential: Optional[str] = None connect_timeout: int = 10000 @@ -79,7 +77,6 @@ def __init__(self, properties: Dict[str, str]): if not self.endpoint: raise ValueError(f"Required property {self.ENDPOINT} is not set") - self.warehouse = properties.get(self.WAREHOUSE) self.auth_token = properties.get(self.AUTH_TOKEN) self.credential = properties.get(self.CREDENTIAL) self.connect_timeout = int(properties.get(self.CONNECT_TIMEOUT, "10000")) @@ -105,11 +102,13 @@ class IcebergNamespace(LanceNamespace): """ Iceberg REST Catalog namespace implementation for Lance. - The prefix (warehouse) is included in the namespace identifier: - - Namespace ID format: [prefix, namespace1, namespace2, ...] - - Table ID format: [prefix, namespace1, namespace2, ..., table_name] + The warehouse is the first element of the namespace/table identifier: + - Namespace ID format: [warehouse, namespace1, namespace2, ...] + - Table ID format: [warehouse, namespace1, namespace2, ..., table_name] - This is consistent with how Polaris handles catalog names. + The implementation caches warehouse -> config mappings by calling + /v1/config?warehouse={warehouse}. If the config contains a prefix, + that prefix is used for API paths; otherwise, the warehouse name is used. """ TABLE_TYPE_LANCE = "lance" @@ -123,8 +122,6 @@ def __init__(self, **properties): headers = {} if self.config.auth_token: headers["Authorization"] = f"Bearer {self.config.auth_token}" - if self.config.warehouse: - headers["X-Iceberg-Access-Delegation"] = "vended-credentials" self.rest_client = RestClient( base_url=self.config.get_base_api_url(), @@ -184,13 +181,13 @@ def _get_prefix_path(self, warehouse: str) -> str: def list_namespaces(self, request: ListNamespacesRequest) -> ListNamespacesResponse: """List namespaces. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Remaining elements specify the parent namespace to list children of. """ ns_id = self._parse_identifier(request.id) if not ns_id: - raise InvalidInputException("Must specify at least the prefix (warehouse)") + raise InvalidInputException("Must specify at least the warehouse") try: prefix = ns_id[0] @@ -231,7 +228,7 @@ def create_namespace( ) -> CreateNamespaceResponse: """Create a new namespace. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Remaining elements are the namespace to create. """ ns_id = self._parse_identifier(request.id) @@ -276,7 +273,7 @@ def describe_namespace( ) -> DescribeNamespaceResponse: """Describe a namespace. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Remaining elements are the namespace to describe. """ ns_id = self._parse_identifier(request.id) @@ -313,7 +310,7 @@ def describe_namespace( def drop_namespace(self, request: DropNamespaceRequest) -> DropNamespaceResponse: """Drop a namespace. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Remaining elements are the namespace to drop. """ if request.behavior and request.behavior.lower() == "cascade": @@ -354,13 +351,13 @@ def drop_namespace(self, request: DropNamespaceRequest) -> DropNamespaceResponse def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: """List tables in a namespace. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Remaining elements are the namespace to list tables from. """ ns_id = self._parse_identifier(request.id) if len(ns_id) < 2: - raise InvalidInputException("Must specify at least prefix and namespace") + raise InvalidInputException("Must specify at least warehouse and namespace") try: prefix = ns_id[0] @@ -404,7 +401,7 @@ def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: """Declare a table (metadata only operation). - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Middle elements are the namespace, last element is the table name. """ table_id = self._parse_identifier(request.id) @@ -464,29 +461,10 @@ def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: except Exception as e: raise InternalException(f"Failed to declare table: {e}") - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only operation). - - .. deprecated:: - Use :meth:`declare_table` instead. - """ - warnings.warn( - "create_empty_table is deprecated, use declare_table instead", - DeprecationWarning, - stacklevel=2, - ) - declare_request = DeclareTableRequest() - declare_request.id = request.id - declare_request.location = request.location - response = self.declare_table(declare_request) - return CreateEmptyTableResponse(location=response.location) - def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse: """Describe a table. - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Middle elements are the namespace, last element is the table name. """ if request.load_detailed_metadata: @@ -546,7 +524,7 @@ def deregister_table( ) -> DeregisterTableResponse: """Deregister a table (remove from catalog without deleting data). - The first element of request.id is treated as the prefix (warehouse). + The first element of request.id is the warehouse. Middle elements are the namespace, last element is the table name. """ table_id = self._parse_identifier(request.id) diff --git a/python/src/lance_namespace_impls/polaris.py b/python/src/lance_namespace_impls/polaris.py index 2dbcc21..4f17c9d 100644 --- a/python/src/lance_namespace_impls/polaris.py +++ b/python/src/lance_namespace_impls/polaris.py @@ -8,10 +8,10 @@ from lance.namespace import LanceNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, - CreateEmptyTableResponse, CreateNamespaceRequest, CreateNamespaceResponse, + DeclareTableRequest, + DeclareTableResponse, DeregisterTableRequest, DeregisterTableResponse, DescribeNamespaceRequest, @@ -282,10 +282,8 @@ def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: except Exception as e: raise InternalException(f"Failed to list tables: {e}") - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only operation).""" + def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: + """Declare a table (metadata only operation).""" table_id = self._parse_identifier(request.id) if len(table_id) < 3: @@ -319,9 +317,9 @@ def create_empty_table( create_request, ) - logger.info(f"Created table: {'.'.join(table_id)}") + logger.info(f"Declared table: {'.'.join(table_id)}") - return CreateEmptyTableResponse(location=table_path) + return DeclareTableResponse(location=table_path) except RestClientException as e: if e.is_conflict(): @@ -332,7 +330,7 @@ def create_empty_table( raise NamespaceNotFoundException( f"Namespace not found: {catalog}.{'.'.join(namespace)}" ) - raise InternalException(f"Failed to create empty table: {e}") + raise InternalException(f"Failed to declare table: {e}") except ( TableAlreadyExistsException, NamespaceNotFoundException, @@ -340,7 +338,7 @@ def create_empty_table( ): raise except Exception as e: - raise InternalException(f"Failed to create empty table: {e}") + raise InternalException(f"Failed to declare table: {e}") def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse: """Describe a table. diff --git a/python/src/lance_namespace_impls/unity.py b/python/src/lance_namespace_impls/unity.py index d70b4b2..503472a 100644 --- a/python/src/lance_namespace_impls/unity.py +++ b/python/src/lance_namespace_impls/unity.py @@ -12,10 +12,10 @@ from lance.namespace import LanceNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, - CreateEmptyTableResponse, CreateNamespaceRequest, CreateNamespaceResponse, + DeclareTableRequest, + DeclareTableResponse, DeregisterTableRequest, DeregisterTableResponse, DescribeNamespaceRequest, @@ -422,10 +422,8 @@ def list_tables(self, request: ListTablesRequest) -> ListTablesResponse: except Exception as e: raise InternalException(f"Failed to list tables: {e}") - def create_empty_table( - self, request: CreateEmptyTableRequest - ) -> CreateEmptyTableResponse: - """Create an empty table (metadata only operation).""" + def declare_table(self, request: DeclareTableRequest) -> DeclareTableResponse: + """Declare a table (metadata only operation).""" table_id = self._parse_identifier(request.id) if len(table_id) != 3: @@ -473,20 +471,20 @@ def create_empty_table( "/tables", create_table, response_converter=_parse_table_info ) - logger.info(f"Created empty table: {catalog}.{schema}.{table}") + logger.info(f"Declared table: {catalog}.{schema}.{table}") - return CreateEmptyTableResponse(location=table_path) + return DeclareTableResponse(location=table_path) except RestClientException as e: if e.is_conflict(): raise TableAlreadyExistsException( f"Table already exists: {'.'.join(request.id)}" ) - raise InternalException(f"Failed to create empty table: {e}") + raise InternalException(f"Failed to declare table: {e}") except Exception as e: if isinstance(e, (TableAlreadyExistsException, InvalidInputException)): raise - raise InternalException(f"Failed to create empty table: {e}") + raise InternalException(f"Failed to declare table: {e}") def describe_table(self, request: DescribeTableRequest) -> DescribeTableResponse: """Describe a table.""" diff --git a/python/tests/test_glue_integration.py b/python/tests/test_glue_integration.py index a6a212a..0fd6762 100644 --- a/python/tests/test_glue_integration.py +++ b/python/tests/test_glue_integration.py @@ -164,7 +164,7 @@ def test_namespace_operations(self): def test_table_operations(self): """Test table CRUD operations.""" from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, + DeclareTableRequest, DescribeTableRequest, DeregisterTableRequest, ListTablesRequest, @@ -174,12 +174,12 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" table_location = f"{self.s3_root}/{db_name}/{table_name}.lance" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [db_name, table_name] create_request.location = table_location - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) self.assertEqual(create_response.location, table_location) @@ -203,7 +203,7 @@ def test_table_operations(self): def test_multiple_tables_in_namespace(self): """Test creating and listing multiple tables in a namespace.""" from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, + DeclareTableRequest, DeregisterTableRequest, ListTablesRequest, ) @@ -213,10 +213,10 @@ def test_multiple_tables_in_namespace(self): for table_name in table_names: table_location = f"{self.s3_root}/{db_name}/{table_name}.lance" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [db_name, table_name] create_request.location = table_location - self.namespace.create_empty_table(create_request) + self.namespace.declare_table(create_request) list_request = ListTablesRequest() list_request.id = [db_name] diff --git a/python/tests/test_hive2.py b/python/tests/test_hive2.py index 5811570..01c9cae 100644 --- a/python/tests/test_hive2.py +++ b/python/tests/test_hive2.py @@ -195,11 +195,10 @@ def test_list_tables(self, hive_namespace, mock_hive_client): mock_client_instance.get_all_tables.assert_called_once_with("test_db") def test_describe_table(self, hive_namespace, mock_hive_client): - """Test describing a table returns location and storage_options only. + """Test describing a table returns location only. Note: load_detailed_metadata=false is the only supported mode, which means - only location and storage_options are returned. Other fields (version, schema, etc.) - are not populated. + only location is returned. Other fields (version, schema, etc.) are not populated. """ mock_table = MagicMock() mock_table.sd.location = "/tmp/warehouse/test_db.db/test_table" @@ -217,10 +216,6 @@ def test_describe_table(self, hive_namespace, mock_hive_client): response = hive_namespace.describe_table(request) assert response.location == "/tmp/warehouse/test_db.db/test_table" - # Only location and storage_options are returned (load_detailed_metadata=false) - assert ( - response.storage_options == {} - ) # Empty since no storage.* properties configured mock_client_instance.get_table.assert_called_once_with("test_db", "test_table") @@ -300,8 +295,6 @@ def test_pickle_support(self): ugi="user:group1,group2", **{ "client.pool-size": "5", - "storage.access_key_id": "test-key", - "storage.secret_access_key": "test-secret", }, ) @@ -315,8 +308,6 @@ def test_pickle_support(self): assert restored.root == "/tmp/warehouse" assert restored.ugi == "user:group1,group2" assert restored.pool_size == 5 - assert restored.storage_properties["access_key_id"] == "test-key" - assert restored.storage_properties["secret_access_key"] == "test-secret" assert restored._client is None diff --git a/python/tests/test_hive2_integration.py b/python/tests/test_hive2_integration.py index 216d713..8203e36 100644 --- a/python/tests/test_hive2_integration.py +++ b/python/tests/test_hive2_integration.py @@ -14,10 +14,10 @@ import pytest -from lance_namespace_impls.hive2 import Hive2Namespace +from lance_namespace_impls.hive2 import Hive2Namespace, HIVE_AVAILABLE from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, CreateNamespaceRequest, + DeclareTableRequest, DeregisterTableRequest, DescribeNamespaceRequest, DescribeTableRequest, @@ -48,7 +48,10 @@ def check_hive_available(): @pytest.mark.integration -@unittest.skipUnless(hive_available, f"Hive2 Metastore is not available at {HIVE_URI}") +@unittest.skipUnless( + HIVE_AVAILABLE and hive_available, + f"Hive2 dependencies not installed or Metastore not available at {HIVE_URI}", +) class TestHive2NamespaceIntegration(unittest.TestCase): """Integration tests for Hive2Namespace against a running Hive2 Metastore.""" @@ -128,12 +131,12 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [self.test_database, table_name] create_request.location = f"/tmp/lance/{self.test_database}/{table_name}" - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) # Describe table @@ -155,19 +158,19 @@ def test_table_operations(self): deregister_request.id = [self.test_database, table_name] self.namespace.deregister_table(deregister_request) - def test_create_empty_table_with_location(self): - """Test creating an empty table with a specific location.""" + def test_declare_table_with_location(self): + """Test declaring a table with a specific location.""" # Create namespace first ns_request = CreateNamespaceRequest() ns_request.id = [self.test_database] self.namespace.create_namespace(ns_request) table_name = "lance_table" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [self.test_database, table_name] create_request.location = f"/tmp/lance/{self.test_database}/{table_name}" - response = self.namespace.create_empty_table(create_request) + response = self.namespace.declare_table(create_request) self.assertIsNotNone(response.location) # Clean up table diff --git a/python/tests/test_hive3.py b/python/tests/test_hive3.py index 9291ffa..d0a235b 100644 --- a/python/tests/test_hive3.py +++ b/python/tests/test_hive3.py @@ -190,11 +190,10 @@ def test_list_tables(self, hive_namespace, mock_hive_client): mock_client_instance.get_all_tables.assert_called_once_with("test_db") def test_describe_table(self, hive_namespace, mock_hive_client): - """Test describing a table returns location and storage_options only. + """Test describing a table returns location only. Note: load_detailed_metadata=false is the only supported mode, which means - only location and storage_options are returned. Other fields (version, schema, etc.) - are not populated. + only location is returned. Other fields (version, schema, etc.) are not populated. """ mock_table = MagicMock() mock_table.sd.location = "/tmp/warehouse/test_db/test_table" @@ -211,10 +210,6 @@ def test_describe_table(self, hive_namespace, mock_hive_client): response = hive_namespace.describe_table(request) assert response.location == "/tmp/warehouse/test_db/test_table" - # Only location and storage_options are returned (load_detailed_metadata=false) - assert ( - response.storage_options == {} - ) # Empty since no storage.* properties configured mock_client_instance.get_table.assert_called_once_with("test_db", "test_table") @@ -263,8 +258,13 @@ def test_normalize_identifier(self, hive_namespace): def test_get_table_location(self, hive_namespace): """Test getting table location for 3-level hierarchy.""" + # Default "hive" catalog uses hive2-compatible path (no catalog in path) location = hive_namespace._get_table_location("hive", "test_db", "test_table") - assert location == "/tmp/warehouse/test_db/test_table.lance" + assert location == "/tmp/warehouse/test_db.db/test_table" + + # Non-default catalog includes catalog in path + location = hive_namespace._get_table_location("custom", "test_db", "test_table") + assert location == "/tmp/warehouse/custom/test_db.db/test_table" def test_root_namespace_operations(self, hive_namespace): """Test root namespace operations.""" @@ -300,8 +300,6 @@ def test_pickle_support(self): ugi="user:group1,group2", **{ "client.pool-size": "5", - "storage.access_key_id": "test-key", - "storage.secret_access_key": "test-secret", }, ) @@ -315,7 +313,6 @@ def test_pickle_support(self): assert restored.root == "/tmp/warehouse" assert restored.ugi == "user:group1,group2" assert restored.pool_size == 5 - assert restored.storage_properties["access_key_id"] == "test-key" assert restored._client is None diff --git a/python/tests/test_hive3_integration.py b/python/tests/test_hive3_integration.py index 80ac949..300ca04 100644 --- a/python/tests/test_hive3_integration.py +++ b/python/tests/test_hive3_integration.py @@ -14,10 +14,10 @@ import pytest -from lance_namespace_impls.hive3 import Hive3Namespace +from lance_namespace_impls.hive3 import Hive3Namespace, HIVE_AVAILABLE from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, CreateNamespaceRequest, + DeclareTableRequest, DeregisterTableRequest, DescribeNamespaceRequest, DescribeTableRequest, @@ -49,7 +49,10 @@ def check_hive_available(): @pytest.mark.integration -@unittest.skipUnless(hive_available, f"Hive3 Metastore is not available at {HIVE_URI}") +@unittest.skipUnless( + HIVE_AVAILABLE and hive_available, + f"Hive3 dependencies not installed or Metastore not available at {HIVE_URI}", +) class TestHive3NamespaceIntegration(unittest.TestCase): """Integration tests for Hive3Namespace against a running Hive3 Metastore.""" @@ -142,12 +145,12 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [self.test_catalog, self.test_database, table_name] create_request.location = f"/tmp/lance/{self.test_database}/{table_name}" - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) # Describe table @@ -169,19 +172,19 @@ def test_table_operations(self): deregister_request.id = [self.test_catalog, self.test_database, table_name] self.namespace.deregister_table(deregister_request) - def test_create_empty_table_with_location(self): - """Test creating an empty table with a specific location.""" + def test_declare_table_with_location(self): + """Test declaring a table with a specific location.""" # Create namespace first ns_request = CreateNamespaceRequest() ns_request.id = [self.test_catalog, self.test_database] self.namespace.create_namespace(ns_request) table_name = "lance_table" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [self.test_catalog, self.test_database, table_name] create_request.location = f"/tmp/lance/{self.test_database}/{table_name}" - response = self.namespace.create_empty_table(create_request) + response = self.namespace.declare_table(create_request) self.assertIsNotNone(response.location) # Clean up table diff --git a/python/tests/test_iceberg.py b/python/tests/test_iceberg.py index f3a1061..b38937d 100644 --- a/python/tests/test_iceberg.py +++ b/python/tests/test_iceberg.py @@ -24,7 +24,7 @@ DescribeNamespaceRequest, DropNamespaceRequest, ListTablesRequest, - CreateEmptyTableRequest, + DeclareTableRequest, DescribeTableRequest, DeregisterTableRequest, ) @@ -39,7 +39,6 @@ def test_config_initialization(self): "endpoint": "https://iceberg.example.com", "root": "/data/lance", "auth_token": "test_token", - "warehouse": "test_warehouse", } config = IcebergNamespaceConfig(properties) @@ -47,7 +46,6 @@ def test_config_initialization(self): self.assertEqual(config.endpoint, "https://iceberg.example.com") self.assertEqual(config.root, "/data/lance") self.assertEqual(config.auth_token, "test_token") - self.assertEqual(config.warehouse, "test_warehouse") def test_config_defaults(self): """Test configuration with default values.""" @@ -59,7 +57,6 @@ def test_config_defaults(self): self.assertEqual(config.root, os.getcwd()) self.assertIsNone(config.auth_token) - self.assertIsNone(config.warehouse) self.assertEqual(config.connect_timeout, 10000) self.assertEqual(config.read_timeout, 30000) self.assertEqual(config.max_retries, 3) @@ -333,8 +330,8 @@ def test_list_tables_invalid_id(self, mock_rest_client_class): namespace.list_tables(request) @patch("lance_namespace_impls.iceberg.RestClient") - def test_create_empty_table(self, mock_rest_client_class): - """Test creating an empty table.""" + def test_declare_table(self, mock_rest_client_class): + """Test declaring a table.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -343,11 +340,11 @@ def test_create_empty_table(self, mock_rest_client_class): namespace = IcebergNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["warehouse1", "test_namespace", "test_table"] request.location = None - response = namespace.create_empty_table(request) + response = namespace.declare_table(request) self.assertEqual( response.location, "/data/lance/warehouse1/test_namespace/test_table" @@ -355,8 +352,8 @@ def test_create_empty_table(self, mock_rest_client_class): mock_client.post.assert_called_once() @patch("lance_namespace_impls.iceberg.RestClient") - def test_create_empty_table_with_location(self, mock_rest_client_class): - """Test creating an empty table with custom location.""" + def test_declare_table_with_location(self, mock_rest_client_class): + """Test declaring a table with custom location.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -365,17 +362,17 @@ def test_create_empty_table_with_location(self, mock_rest_client_class): namespace = IcebergNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["warehouse1", "test_namespace", "test_table"] request.location = "/custom/path/test_table" - response = namespace.create_empty_table(request) + response = namespace.declare_table(request) self.assertEqual(response.location, "/custom/path/test_table") @patch("lance_namespace_impls.iceberg.RestClient") - def test_create_empty_table_already_exists(self, mock_rest_client_class): - """Test creating a table that already exists.""" + def test_declare_table_already_exists(self, mock_rest_client_class): + """Test declaring a table that already exists.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -386,25 +383,25 @@ def test_create_empty_table_already_exists(self, mock_rest_client_class): namespace = IcebergNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["warehouse1", "test_namespace", "existing_table"] with self.assertRaises(TableAlreadyExistsException): - namespace.create_empty_table(request) + namespace.declare_table(request) @patch("lance_namespace_impls.iceberg.RestClient") - def test_create_empty_table_invalid_id(self, mock_rest_client_class): - """Test creating table with invalid ID fails.""" + def test_declare_table_invalid_id(self, mock_rest_client_class): + """Test declaring table with invalid ID fails.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client namespace = IcebergNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["warehouse1", "only_namespace"] with self.assertRaises(InvalidInputException): - namespace.create_empty_table(request) + namespace.declare_table(request) @patch("lance_namespace_impls.iceberg.RestClient") def test_describe_table(self, mock_rest_client_class): diff --git a/python/tests/test_iceberg_integration.py b/python/tests/test_iceberg_integration.py index b31c1e2..83af4e5 100644 --- a/python/tests/test_iceberg_integration.py +++ b/python/tests/test_iceberg_integration.py @@ -18,8 +18,8 @@ from lance_namespace_impls.iceberg import IcebergNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, CreateNamespaceRequest, + DeclareTableRequest, DeregisterTableRequest, DescribeNamespaceRequest, DescribeTableRequest, @@ -120,12 +120,12 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [self.test_warehouse, self.test_namespace, table_name] create_request.location = f"s3://warehouse/{self.test_namespace}/{table_name}" - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) # Describe table @@ -147,19 +147,19 @@ def test_table_operations(self): deregister_request.id = [self.test_warehouse, self.test_namespace, table_name] self.namespace.deregister_table(deregister_request) - def test_create_empty_table_with_location(self): - """Test creating an empty table with a specific location.""" + def test_declare_table_with_location(self): + """Test declaring a table with a specific location.""" # Create namespace first ns_request = CreateNamespaceRequest() ns_request.id = [self.test_warehouse, self.test_namespace] self.namespace.create_namespace(ns_request) table_name = "lance_table" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [self.test_warehouse, self.test_namespace, table_name] create_request.location = f"s3://warehouse/{self.test_namespace}/{table_name}" - response = self.namespace.create_empty_table(create_request) + response = self.namespace.declare_table(create_request) self.assertIsNotNone(response.location) # Clean up table diff --git a/python/tests/test_polaris.py b/python/tests/test_polaris.py index 3e2d81f..3ed7130 100644 --- a/python/tests/test_polaris.py +++ b/python/tests/test_polaris.py @@ -23,7 +23,7 @@ DescribeNamespaceRequest, DropNamespaceRequest, ListTablesRequest, - CreateEmptyTableRequest, + DeclareTableRequest, DescribeTableRequest, DeregisterTableRequest, ) @@ -287,8 +287,8 @@ def test_list_tables(self, mock_rest_client_class): ) @patch("lance_namespace_impls.polaris.RestClient") - def test_create_empty_table(self, mock_rest_client_class): - """Test creating an empty table.""" + def test_declare_table(self, mock_rest_client_class): + """Test declaring a table.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -296,11 +296,11 @@ def test_create_empty_table(self, mock_rest_client_class): namespace = PolarisNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["test_catalog", "test_namespace", "test_table"] request.location = None - response = namespace.create_empty_table(request) + response = namespace.declare_table(request) self.assertEqual( response.location, "/data/lance/test_catalog/test_namespace/test_table" @@ -308,8 +308,8 @@ def test_create_empty_table(self, mock_rest_client_class): mock_client.post.assert_called_once() @patch("lance_namespace_impls.polaris.RestClient") - def test_create_empty_table_with_location(self, mock_rest_client_class): - """Test creating an empty table with custom location.""" + def test_declare_table_with_location(self, mock_rest_client_class): + """Test declaring a table with custom location.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -317,11 +317,11 @@ def test_create_empty_table_with_location(self, mock_rest_client_class): namespace = PolarisNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["test_catalog", "test_namespace", "test_table"] request.location = "/custom/path/test_table" - response = namespace.create_empty_table(request) + response = namespace.declare_table(request) self.assertEqual(response.location, "/custom/path/test_table") mock_client.post.assert_called_once_with( @@ -335,8 +335,8 @@ def test_create_empty_table_with_location(self, mock_rest_client_class): ) @patch("lance_namespace_impls.polaris.RestClient") - def test_create_empty_table_already_exists(self, mock_rest_client_class): - """Test creating a table that already exists.""" + def test_declare_table_already_exists(self, mock_rest_client_class): + """Test declaring a table that already exists.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -346,11 +346,11 @@ def test_create_empty_table_already_exists(self, mock_rest_client_class): namespace = PolarisNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["test_catalog", "test_namespace", "existing_table"] with self.assertRaises(TableAlreadyExistsException): - namespace.create_empty_table(request) + namespace.declare_table(request) @patch("lance_namespace_impls.polaris.RestClient") def test_describe_table(self, mock_rest_client_class): @@ -477,11 +477,11 @@ def test_invalid_table_id(self): """Test that table operations fail with invalid identifiers.""" namespace = PolarisNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["catalog", "only_namespace"] # Missing table name with self.assertRaises(InvalidInputException): - namespace.create_empty_table(request) + namespace.declare_table(request) def test_invalid_namespace_id(self): """Test that namespace operations fail with invalid identifiers.""" diff --git a/python/tests/test_polaris_integration.py b/python/tests/test_polaris_integration.py index af35f06..786feeb 100644 --- a/python/tests/test_polaris_integration.py +++ b/python/tests/test_polaris_integration.py @@ -17,8 +17,8 @@ from lance_namespace_impls.polaris import PolarisNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, CreateNamespaceRequest, + DeclareTableRequest, DeregisterTableRequest, DescribeNamespaceRequest, DescribeTableRequest, @@ -152,12 +152,12 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [self.test_catalog, self.test_namespace, table_name] create_request.location = f"/data/warehouse/{self.test_namespace}/{table_name}" - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) # Describe table @@ -179,19 +179,19 @@ def test_table_operations(self): deregister_request.id = [self.test_catalog, self.test_namespace, table_name] self.namespace.deregister_table(deregister_request) - def test_create_empty_table_with_location(self): - """Test creating an empty table with a specific location.""" + def test_declare_table_with_location(self): + """Test declaring a table with a specific location.""" # Create namespace first ns_request = CreateNamespaceRequest() ns_request.id = [self.test_catalog, self.test_namespace] self.namespace.create_namespace(ns_request) table_name = "lance_table" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [self.test_catalog, self.test_namespace, table_name] create_request.location = f"/data/warehouse/{self.test_namespace}/{table_name}" - response = self.namespace.create_empty_table(create_request) + response = self.namespace.declare_table(create_request) self.assertIsNotNone(response.location) # Clean up table diff --git a/python/tests/test_unity.py b/python/tests/test_unity.py index 4a9f5e0..fc6e0dc 100644 --- a/python/tests/test_unity.py +++ b/python/tests/test_unity.py @@ -23,7 +23,7 @@ DescribeNamespaceRequest, DropNamespaceRequest, ListTablesRequest, - CreateEmptyTableRequest, + DeclareTableRequest, DescribeTableRequest, ) @@ -302,8 +302,8 @@ def test_list_tables(self, mock_rest_client_class): self.assertEqual(sorted(response.tables), ["table1", "table3"]) @patch("lance_namespace_impls.unity.RestClient") - def test_create_empty_table(self, mock_rest_client_class): - """Test creating an empty table.""" + def test_declare_table(self, mock_rest_client_class): + """Test declaring a table.""" mock_client = MagicMock() mock_rest_client_class.return_value = mock_client @@ -321,10 +321,10 @@ def test_create_empty_table(self, mock_rest_client_class): namespace = UnityNamespace(**self.properties) - request = CreateEmptyTableRequest() + request = DeclareTableRequest() request.id = ["test_catalog", "test_schema", "test_table"] - response = namespace.create_empty_table(request) + response = namespace.declare_table(request) self.assertEqual( response.location, "/data/lance/test_catalog/test_schema/test_table" diff --git a/python/tests/test_unity_integration.py b/python/tests/test_unity_integration.py index 31f99c4..8ee3619 100644 --- a/python/tests/test_unity_integration.py +++ b/python/tests/test_unity_integration.py @@ -17,8 +17,8 @@ from lance_namespace_impls.unity import UnityNamespace from lance_namespace_urllib3_client.models import ( - CreateEmptyTableRequest, CreateNamespaceRequest, + DeclareTableRequest, DeregisterTableRequest, DescribeNamespaceRequest, DescribeTableRequest, @@ -147,14 +147,14 @@ def test_table_operations(self): table_name = f"test_table_{uuid.uuid4().hex[:8]}" - # Create empty table (DeclareTable) - create_request = CreateEmptyTableRequest() + # Declare table + create_request = DeclareTableRequest() create_request.id = [UNITY_CATALOG, self.test_schema, table_name] create_request.location = ( f"/tmp/lance/{UNITY_CATALOG}/{self.test_schema}/{table_name}" ) - create_response = self.namespace.create_empty_table(create_request) + create_response = self.namespace.declare_table(create_request) self.assertIsNotNone(create_response.location) # Describe table @@ -176,21 +176,21 @@ def test_table_operations(self): deregister_request.id = [UNITY_CATALOG, self.test_schema, table_name] self.namespace.deregister_table(deregister_request) - def test_create_empty_table_with_location(self): - """Test creating an empty table with a specific location.""" + def test_declare_table_with_location(self): + """Test declaring a table with a specific location.""" # Create namespace first ns_request = CreateNamespaceRequest() ns_request.id = [UNITY_CATALOG, self.test_schema] self.namespace.create_namespace(ns_request) table_name = "lance_table" - create_request = CreateEmptyTableRequest() + create_request = DeclareTableRequest() create_request.id = [UNITY_CATALOG, self.test_schema, table_name] create_request.location = ( f"/tmp/lance/{UNITY_CATALOG}/{self.test_schema}/{table_name}" ) - response = self.namespace.create_empty_table(create_request) + response = self.namespace.declare_table(create_request) self.assertIsNotNone(response.location) # Clean up table diff --git a/python/uv.lock b/python/uv.lock index fad9ad7..5d832e9 100644 --- a/python/uv.lock +++ b/python/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "lance-namespace-impls" -version = "0.0.1" +version = "0.1.0" source = { editable = "." } dependencies = [ { name = "lance-namespace-urllib3-client" },