Question: Schema evolution interfaces vs. namespace impls — design boundary clarification

[FANNG1]

Context

After studying the Lance Namespace spec, the implementations in lance-namespace-impls,
and how engines (lance-spark, lance-ray, Daft, LanceDB) consume the interfaces, I'd like
to clarify the design intent around schema-related operations and their role in the
namespace architecture.

Observations

1. External catalog impls don't treat Lance schema as authoritative catalog state

External catalog impls (Unity, Glue, Hive, Polaris, Iceberg in lance-namespace-impls)
handle metadata at different levels:

• Location: all impls persist table location as core catalog state
• Table properties: some impls sync properties to catalog (e.g. Unity stores them in
  properties map, Glue in Parameters)
• Schema: Unity stores columns converted from Lance Arrow schema, but this is a
  registration-time snapshot — it's not kept in sync with the actual dataset schema after
  subsequent schema evolution operations
• load_detailed_metadata=true: external impls (Hive2, Iceberg, etc.) explicitly reject
  this with "not supported", meaning they cannot return current schema/stats/version from the
  catalog — these can only be obtained by opening the Lance dataset

So while some catalog metadata is persisted, the Lance dataset on storage remains the
single source of truth for current schema state.

2. Schema evolution interfaces: spec defines them, but adoption is minimal

The spec defines AlterTableAddColumns, AlterTableAlterColumns, and
AlterTableDropColumns. The current implementation status:

• Core trait: defaults to not_supported error (lance-namespace/src/namespace.rs:399)
• REST client: has HTTP forwarding/route definitions (lance-namespace-impls/src/rest.rs:1436)
• External catalog impls (Unity, Glue, Hive, Polaris, Iceberg): none implement these
• Engine side: none of lance-spark, lance-ray, Daft, or LanceDB call these namespace
  interfaces — all schema changes go through the Lance SDK directly on storage

3. Engine-side usage pattern

From analyzing lance-spark, lance-ray, Daft, and LanceDB source code:

• Namespace interfaces used by engines: primarily metadata (Describe/Declare/List/Drop
  Table, Namespace CRUD) plus credential vending
• DML via namespace: only Daft (REST path) and LanceDB (QueryTable/CreateTable pushdown)
• Schema evolution, index management via namespace: none — all four engines go through
  the Lance SDK directly

Questions

1. Schema as catalog state: Is the current "dataset on storage is the sole authority for
   schema" the intended long-term model? Or is there a plan for catalog impls to maintain
   synchronized schema state (similar to how Iceberg catalogs track current schema)?

2. Schema evolution interfaces: These interfaces exist in the spec and REST client has
   forwarding routes, but no external catalog impl has adopted them and no engine calls them.
   Are they reserved for future REST-server-side use cases? Or is there a plan for external
   impls to implement them as SDK pass-through?

3. Engine adoption: Is there a roadmap for engines to start routing schema evolution /
   index management through namespace interfaces rather than SDK-direct? Understanding this
   helps clarify whether new namespace implementations should prioritize these interfaces.

[FANNG1]

cc @jackye1995 @Xuanwo

[Xuanwo]

Thanks for the detailed investigation. I think your reading is correct.

The current intended model is that the Lance dataset on storage remains the source of truth for schema/version/stats. Namespace implementations are not required to maintain a synchronized catalog-side schema. For basic catalog implementations, DescribeTable only needs to return location/storage options with load_detailed_metadata=false; load_detailed_metadata=true means the implementation has to open the Lance dataset and read the actual metadata.

The schema evolution APIs in the namespace spec should be treated as optional data-plane/server-side capabilities, mainly useful for REST namespace or managed service implementations. External catalog integrations such as Glue/Hive/Unity/Polaris should not be expected to implement them unless they can clearly define the consistency boundary between the Lance dataset commit and catalog metadata updates.

So for new namespace implementations, I would prioritize the basic metadata operations first: namespace/table CRUD, DeclareTable, DescribeTable(load_detailed_metadata=false), storage options/credential vending where applicable. Schema evolution and index operations can be added later only if the implementation intentionally supports server-side dataset mutations.

We should probably make this clearer in the spec/docs, especially the distinction between catalog metadata, detailed dataset metadata, and optional data-plane operations.

[FANNG1]

Thanks for the clarification, this makes sense.

One follow-up concern is that different namespace implementations may support different optional capabilities and consistency models. In practice, clients may need to adapt their behavior depending on what a specific REST namespace implementation actually supports.

To make that easier, I opened #342 to propose a capability discovery endpoint for Lance REST Namespace. That would let clients inspect the server-reported capabilities and choose compatible code paths or fallback behavior, so different namespace impls can interoperate more smoothly under the same client abstraction.

I think that would complement the spec/docs clarification here nicely.