feat: support per-base storage options with base_<id>.<key> prefix#7608
Merged
jackye1995 merged 2 commits intoJul 3, 2026
Merged
Conversation
Multi-base datasets previously could only receive per-base credentials through ObjectStoreParams bindings keyed by base path URI, which cannot be expressed in the flat storage_options map used by the bindings and namespace credential vending. A storage option key of the form base_<id>.<key> now applies <key> only to the registered base path with that manifest id. Unscoped options are shared defaults inherited by every base; scoped entries override them per base. The primary dataset store strips all scoped entries. When options come from a dynamic provider, per-base options are re-resolved on every refresh through the parent accessor. Exact per-URI base_store_params bindings keep precedence over scoped keys.
github-actions
Bot
added
A-python
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Provider-backed accessors now always resolve base scope per fetch, so per-base credentials vended by a provider apply even without initial options; a scope_resolved flag keeps re-scoping idempotent. Forced refreshes propagate through the wrapper to the origin provider via force_fetch_storage_options. The parent accessor's cache expiry is the minimum of the unscoped and all base-scoped expires_at_millis entries, so the earliest-expiring credential drives refresh. Unregistered scoped ids log at warn on the write path.
BubbleCal
approved these changes
Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Multi-base datasets can place data on different buckets or accounts, but per-base credentials
could only be passed through
base_store_params(ObjectStoreParamskeyed by base path URI).The flat
storage_optionsmap used by the Python/Java bindings and namespace credential vendingcannot express per-base credentials, and per-URI bindings are static so they never compose with
dynamic credential refresh.
Change
A storage option key of the form
base_<id>.<key>now applies<key>only to the registeredbase path with manifest id
<id>:them per base. Example:
{"account_key": "shared", "base_1.account_key": "abc"}givesbase 1
account_key = abcwhile everything else is shared.ObjectStoreRegistry::get_storechoke point (no-op when no scoped keys are present, preservingstore cache identity).
refresh through the parent accessor (
BaseScopedStorageOptionsProvider), so a namespace servercan vend per-base credentials in one flat map and refresh works per base.
base_store_paramsbinding, thenbase_<id>.<key>overlay,then shared defaults.
runtime-only; nothing is persisted in the manifest.
Resolution happens in the Rust core (
Dataset::store_params_for_base, write-path target-baseresolution, external blob base resolver), so Python and Java get the feature through the existing
storage_optionsparameter without API changes. Documented in the object store guide and therelevant API docs.