Skip to content

fix(index): defer scalar type validation to pylance#5219

Open
FANNG1 wants to merge 1 commit into
lance-format:mainfrom
FANNG1:fix/defer-scalar-type-validation
Open

fix(index): defer scalar type validation to pylance#5219
FANNG1 wants to merge 1 commit into
lance-format:mainfrom
FANNG1:fix/defer-scalar-type-validation

Conversation

@FANNG1

@FANNG1 FANNG1 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Remove lance-ray's driver-side Arrow type whitelist for distributed BTREE scalar indexes
  • Keep strict string validation for INVERTED / FTS, where the distributed layer has a clear semantic requirement
  • Let pylance own scalar index type compatibility for BTREE and other scalar index builders reached through create_index_uncommitted
  • Add a driver-level regression test proving pa.large_string() reaches the distributed segment workflow instead of being rejected by lance-ray first

Background

lance-format/lance#7525 added large_string / LargeUtf8 support for scalar indexes in pylance. Before this change, lance-ray could still reject the same column earlier because it mirrored a narrower Arrow type matrix in create_scalar_index.

Keeping a duplicate whitelist here means lance-ray can fall behind pylance whenever scalar index type support expands. The distributed layer still validates distributed workflow concerns such as supported distributed index types, column resolution, worker count, fragment IDs, and namespace/URI inputs.

Verification

uv run --no-sync pytest \
  tests/test_vector_index_options.py::test_create_scalar_index_defers_large_string_scalar_validation \
  tests/test_vector_index_options.py::test_create_scalar_index_uses_segment_path -q
# 5 passed

uv run --no-sync ruff check lance_ray/index.py tests/test_vector_index_options.py
# All checks passed!

@github-actions github-actions Bot added the bug Something isn't working label Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant