fix(spark): propagate per-read .option() storage credentials into path-based loadTable

[sezruby]

Problem

Path-based reads that pass storage credentials via the DataFrame .option(...) API
(e.g. an Azure SAS token) fail with an Azure Managed Identity (IMDS 169.254.169.254)
token error. Fixes #665.

LanceDataSource implements SupportsCatalogOptions, so Spark calls
extractIdentifier(options) → catalog.loadTable(identifier) and forwards only the
Identifier. LanceIdentifier kept just the location, so the path-based opens rebuilt
storage options from catalogConfig alone — empty for per-read credentials — and the
native open fell back to object_store's default Azure credential chain (MSI).

Change

• LanceIdentifier gains an optional immutable options map (back-compat 1-arg
  constructor retained). On the load path these options override catalog-level
  storage.* defaults on conflict; catalog options still fill gaps.
• LanceDataSource.extractIdentifier captures the reader options onto the identifier.
• Utils.createReadOptions gets an overload taking pathOptions, seeded before
  withCatalogDefaults to get that precedence. The dataset-URI key (path) is stripped
  from the seed so it does not leak into the native storage_options map (related to
  fix(spark): strip recognized typed options from Rust storage_options map #520, which removes the other recognized typed keys).
• Both path-based opens are covered, via a shared getPathOptions(Identifier) helper:
  • loadTableFromPath (the time-travel resolution open and the main open), and
  • tableExistsAtPath, which had the identical drop → MSI bug and is on the
    CREATE TABLE IF NOT EXISTS / overwrite path.

Credential precedence on path-based access: reader .option(...) > catalog
storage.* defaults.

Tests

• LanceIdentifierOptionsTest (unit, base module): identifier carries options
  immutably; null → empty; per-read options reach getStorageOptions(); per-read
  overrides catalog default while catalog default still fills gaps; the path key is
  not present in the storage options.
• AbfssSasReadIntegrationTest (3.5 module, @EnabledIfEnvironmentVariable-gated so CI
  skips it without creds): reads a real abfss:// dataset with the SAS supplied only via
  .option(...). Verified locally — fails on main with the MSI error
  (GET http://169.254.169.254/.../oauth2/token), passes with this change. The test
  reads the SAS from a neutral env var (LANCE_IT_SAS) so lance core's AZURE_STORAGE_*
  env fallback can't mask the connector behavior. Relates to Run test against Azure #241.

Notes

• Complementary to fix(catalog): propagate namespace storage options into read options #522 (namespace/REST-vended creds) and fix(spark): strip recognized typed options from Rust storage_options map #520 (typed-key stripping); no
  functional overlap. If fix(catalog): propagate namespace storage options into read options #522 merges first, the path-based call sites become
  createPathBasedReadOptions(...); this patch rebases by adding the pathOptions
  overload there instead — trivial.

🤖 Generated with Claude Code