[Tracking] Lance feature matrix for Doris 4.2

[Gabriel39]

Goal

Plan and track the Doris 4.2 support scope for Lance Catalog, Lance dataset access, and vector search.

Doris 4.2 work items

Check an item only after implementation, tests, documentation, and verification are complete.

Tracking convention (2026-09-30): completion on branch-4.1 counts as completion for this Doris 4.2 feature matrix; a separate branch-4.2 merge is not a tracking prerequisite. This records implementation delivery, not availability in a published release.

Catalog integration

• Filesystem Lance Catalog with a local path
• Filesystem Lance Catalog with file://
• Filesystem Lance Catalog with s3://
• Lance REST Namespace Catalog

Authentication and credentials

• Bearer Token authentication
• API Key authentication
• Custom HTTP headers
• Temporary storage credentials returned by the Namespace service

Namespace and metadata

• Map multilevel Lance Namespaces to Doris databases
• SHOW DATABASES
• SHOW TABLES
• DESC

Namespace and table DDL

The baseline implementation in #65730 is read-only. SHOW DATABASES, SHOW TABLES, and DESC are supported through ListNamespaces, ListTables, and DescribeTable, while LanceExternalCatalog does not currently provide ExternalMetadataOps or invoke namespace/table mutation APIs. The remaining DDL work below is planned for Doris 4.2 and tracked by #66516.

• CREATE DATABASE / Lance Namespace
• DROP DATABASE / Lance Namespace
• CREATE TABLE / Lance dataset and catalog registration
• SHOW CREATE TABLE
• ALTER TABLE schema evolution — Add, alter/backfill, rename, and drop columns where supported by Lance.
• RENAME TABLE
• DROP TABLE / deregister table — Define whether each operation deletes the underlying dataset or only removes the catalog entry.
• Metadata cache refresh and failure semantics for DDL

Versioning and consistency

• Pin the dataset version during planning — Preserve snapshot consistency for the full query lifecycle.
• REST Namespace managed versioning in the BE reader — [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453 and [docs](lance) Document time travel, tags, branches and managed versioning doris-website#4172; FE resolves the selected snapshot for the existing BE reader.
• Time travel and explicit version selection for catalog scans — [Feature](lance) Support time travel and REST managed versioning #66491 / [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453; search TVF snapshot selection remains separate in [Feature](lance) Support snapshot selection in vector and full-text search #68625.

Scan execution

• Lance reader in File Scanner V2
• Parallel scan by Lance Fragment
• Column pruning
• Retain unsupported predicates for execution in Doris — Keep predicates in Doris whenever pushdown cannot preserve SQL semantics.

Predicate pushdown

• Push semantically compatible scalar predicates to Lance — Push only predicates whose semantics are compatible with Lance.
• SQL filter for vector_search()
• Substrait filter for regular Lance scans

Observability

• Show pushed predicates in EXPLAIN — Expose pushdown decisions for troubleshooting and verification.

Direct dataset access

• Read a Lance dataset through the s3() TVF
• Read a Lance dataset through the local() TVF

Paths and splits

• Split one Lance dataset into multiple Fragment splits — Generate parallel scan splits from Lance Fragments.
• Expand glob patterns to multiple Lance datasets — Resolve one path pattern into multiple datasets.
• Read multiple Lance datasets in one TVF invocation — Plan and scan more than one dataset in a single TVF call.
• Support additional file TVF entry points and broader path handling

Read operations

• Read Lance datasets

Search modes

• ANN index search
• Flat exact search
• Radius/Range Search — Expose lower and upper distance bounds.
• Full-text search with BM25 — Expose standalone BM25 full-text search.
• Multi-vector Search — Support multi-vector columns and multiple sub-vectors in one logical query.

Vector index algorithms

• IVF_FLAT query support
• IVF_SQ query support
• IVF_PQ query support
• IVF_HNSW_FLAT query support
• IVF_HNSW_SQ query support
• IVF_HNSW_PQ query support

Distance metrics

• L2
• Cosine
• Dot Product
• Hamming — Support compatible packed binary vectors and indexes.

Data types

• FixedSizeList
• FixedSizeList
• FixedSizeList — Define compatible ANN index and distance-metric combinations.
• FixedSizeList — Support UInt8 and compatible packed binary-vector use cases.
• FixedSizeList
• List<FixedSizeList<...>> for multi-vector columns — Map nested vector columns and define multi-vector scoring semantics.
• Blob v2
• Arrow JSON Extension
• BFloat16 Extension
• Arrow null
• Arrow duration

Query input

• One query vector per search
• Multiple sub-vectors in one logical query

Filtering

• Scalar pre-filter — Apply scalar predicates before vector candidate selection.
• Scalar post-filter — Allow scalar predicates to run after vector candidate selection.

Index lifecycle

• Create vector indexes
• Create scalar indexes
• Show the number of indexes — Exact logical-index counts remain deferred in the accepted [Feature](lance) Add vector and scalar index lifecycle management #66497 inspection contract.
• Show logical index names, columns, and types for Directory catalogs — SHOW INDEX, [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 and [doc](lance) add doc about SHOW INDEX for lance catalog doris-website#4090.
• Show index names, UUIDs, columns, types, and dataset versions — Code and tests are merged in [feature](lance) Support SHOW INDEX for Lance Directory tables #66637 / [feature](lance) Add lance_index_entries inspection TVF #66671; user documentation for the physical lance_index_entries TVF remains outstanding. REST inspection is outside the delivered scope.
• Replace or rebuild an index with the same name
• Drop indexes by name

Index selection

• Automatically select an index for the target vector column — Use Lance index selection for the requested vector column.

ANN tuning

• nprobes
• refine_factor
• ef
• Disable the ANN index with use_index=false

Result output

• Top-K
• Offset
• Return distance through the _distance virtual column — Document exact, approximate, and refined distance semantics.

Tests and documentation

• FE unit tests
• BE unit tests
• Regression tests for Catalog queries, predicate pushdown, file TVFs, and vector search
• Lance Catalog user documentation — Publish the supported scope, configuration, examples, and limitations.

Explicitly not planned for Doris 4.2

These capabilities are excluded from the Doris 4.2 plan.

Category	Capability	Reason
Read and write operations	🚫 INSERT into Lance	Doris 4.2 does not include a Lance write path for appending rows.
Read and write operations	🚫 UPDATE Lance data	Doris 4.2 does not include a Lance write path for modifying and committing rows.
Read and write operations	🚫 DELETE Lance data	Doris 4.2 does not include a Lance write path for deleting and committing rows.
Search modes	🚫 Hybrid Search (Vector + BM25)	LanceDB can orchestrate two searches at a higher layer, but Lance Scanner/lance-c makes nearest and FTS mutually exclusive.
Search modes	🚫 Batch Search	Lance Rust supports independent batch queries; the lance-c binding and Doris per-query Top-K integration are still missing, tracked in #68394. This follow-up does not change the release scope here.
Search modes	🚫 Joint search and scoring across multiple vector columns	One Lance vector query accepts only one vector column.
Distance metrics	🚫 Custom distance functions	Lance ANN accepts only its built-in distance enum and cannot accept a callback, SQL UDF, or arbitrary expression.
Data types	🚫 Vector element types outside the current nearest C ABI	The current nearest C ABI defines Float16, Float32, Float64, UInt8, and Int8 only.
Index selection	🚫 Select index_name at query time	Lance Scanner Query and lance-c nearest do not expose an index-name parameter.
Index selection	🚫 Select index_type at query time	The current query path does not expose an index-type selector.

Completion criteria

• The capability is merged into branch-4.1; this satisfies branch completion for this Doris 4.2 tracker.
• FE/BE unit tests and regression tests cover the capability.
• User documentation describes the supported behavior, parameter semantics, and limitations.
• The capability is verified with representative Lance datasets and object-storage access where applicable.

Sub-issue status audit — 2026-09-30

Checked all ten direct sub-issues; none has nested sub-issues. Status below follows the branch-4.1 completion convention above. This was an audit of merged changes, documented validation, and existing CI results, not a new test run.

Sub-issue	Status	Evidence / remaining work
#66491 — time travel and REST managed versioning	Completed; remains closed	#68453 and apache/doris-website#4172 merged. Catalog scans only; search TVF snapshot selection is #68625. The disabled managed-versioning unit-test follow-up recorded in #66491 remains outstanding.
#66492 — file TVF paths and multiple datasets	Open	No completed glob/multi-dataset implementation was identified.
#66493 — Radius/Range and full-text search	Partially completed; open	Standalone BM25: #67289 and apache/doris-website#4112 merged. Radius/Range still lacks Doris integration; lance-format/lance-c#90 is an open upstream binding PR. Distributed BM25 global statistics remain #67435.
#66494 — multi-vector columns and queries	Completed; remains closed	#68028 and apache/doris-website#4148 merged; checklist synchronized.
#66495 — additional ANN algorithms	Completed; closed in this audit	#66779, #66512, #67039 and apache/doris-website#4082 provide physical-index fixtures, compatibility coverage, and documentation.
#66496 — additional data types	Partially completed; open	JSON, BFloat16, Null, Duration: #67325 / #67921 and apache/doris-website#4182. Blob v2 remains unsupported.
#66497 — index lifecycle	Partially completed; open	Read-only inspection merged in #66637 / #66671; logical inspection documented in apache/doris-website#4090. Physical-TVF documentation and enabled end-to-end mutations remain unfinished; #67978, #67754, #68668, #68669 are still open.
#66516 — namespace and table DDL	In progress; open	Implementation #68349 and documentation apache/doris-website#4173 are both open.
#67060 — time travel and explicit version selection	Completed; closed in this audit	Empty duplicate work item covered by #66491 / #68453 and apache/doris-website#4172.
#68394 — independent batch vector search	Open	C binding and Doris integration remain unfinished; multi-vector support does not implement per-query batch Top-K.

ANN coverage is bounded by the documented algorithm/type/metric compatibility matrix; checking an algorithm does not assert that every possible combination is supported. Multi-vector scoring and limits remain those documented in #68028 / apache/doris-website#4148.

[FANNG1]

Great to see Lance support moving forward, and thanks for laying the scope out this explicitly — the "explicitly not planned" table with the reasoning behind each exclusion is especially useful for setting expectations.

I'm interested in helping out here. Once the work items are broken down into individual issues, I'd be happy to pick some up — feel free to ping me.

[Gabriel39]

Hi @FANNG1, we're very happy to see your interest! We already have an initial PR in progress that covers the basic code framework and some foundational query functionality. We expect it to be merged next week. After that, we'll break the remaining work down into subtasks and share them with the community. We'll be sure to ping you then—thanks for offering to help!

[u70b3]

Hi @Gabriel39, excited to see Lance support coming to Doris! I'd like to help with the community subtasks once #65730 lands.

Some context: I've been working on the FE external-catalog metadata path recently — I collaborated on #66080 (avoid loading every table for SHOW TABLES on external catalogs) and authored #66452 (fix the 1+N remote table enumeration in bypass-mode getTables()). So I'm familiar with ExternalCatalog/ExternalDatabase, remote→local name mapping, and the metadata cache layer.

I'd be happy to pick up any of these, roughly in order of preference:

1. Namespace and metadata: multi-level namespace → Doris db mapping, SHOW DATABASES / SHOW TABLES / DESC support
2. Catalog integration: Filesystem / REST namespace catalog pieces
3. Tests: FE unit tests / regression tests for the Lance catalog

Feel free to ping me when the subtasks are split — thanks!

[morningman]

Hi @Gabriel39, excited to see Lance support coming to Doris! I'd like to help with the community subtasks once #65730 lands.

Some context: I've been working on the FE external-catalog metadata path recently — I collaborated on #66080 (avoid loading every table for SHOW TABLES on external catalogs) and authored #66452 (fix the 1+N remote table enumeration in bypass-mode getTables()). So I'm familiar with ExternalCatalog/ExternalDatabase, remote→local name mapping, and the metadata cache layer.

I'd be happy to pick up any of these, roughly in order of preference:

1. Namespace and metadata: multi-level namespace → Doris db mapping, SHOW DATABASES / SHOW TABLES / DESC support
2. Catalog integration: Filesystem / REST namespace catalog pieces
3. Tests: FE unit tests / regression tests for the Lance catalog

Feel free to ping me when the subtasks are split — thanks!

Thank you for your interest, we may release a set of subtasks once the first PR merged!