Summary
Dataset::deep_clone cannot clone a dataset into a **different object store /****
**account (e.g. from one Azure storage account to another). The copy of the data
files works, but the commit fails because the Operation::Clone handler reads the
source manifest through the destination object store, which cannot see the
source.
Repro (conceptual)
// source dataset opened with account-A credentials
let mut src = Dataset::open("abfss://fs@accountA.dfs.core.windows.net/ds").await?;
// clone into account B, supplying account-B credentials as store_params
src.deep_clone("abfss://fs@accountB.dfs.core.windows.net/ds_copy", version, Some(accountB_params)).await?;
Observed
The commit step fails resolving/reading the source manifest, e.g.:
DatasetNotFound { path: "<source>/_versions/N.manifest", ... }
because do_commit_new_dataset (the Operation::Clone arm) calls
resolve_version_location / load_manifest and reads the index section through
object_store — the destination store — rather than the source store.
Root cause
deep_clone copies files source → target correctly, but the new manifest is built
during commit by reading the source manifest. That read uses the destination
store. Same-store clones work only because the two stores coincide; cross-store
clones cannot find the source.
Proposed fix
Thread the source dataset's object store into the commit path and use it to read
the source manifest (CommitBuilder::with_source_store), while the new dataset is
written through the destination store. Stream files source → target when the
stores differ; keep the server-side copy fast-path when they match. PR to follow.
Notes
Summary
Dataset::deep_clonecannot clone a dataset into a **different object store /******account (e.g. from one Azure storage account to another). The copy of the data
files works, but the commit fails because the
Operation::Clonehandler reads thesource manifest through the destination object store, which cannot see the
source.
Repro (conceptual)
Observed
The commit step fails resolving/reading the source manifest, e.g.:
because
do_commit_new_dataset(theOperation::Clonearm) callsresolve_version_location/load_manifestand reads the index section throughobject_store— the destination store — rather than the source store.Root cause
deep_clonecopies files source → target correctly, but the new manifest is builtduring commit by reading the source manifest. That read uses the destination
store. Same-store clones work only because the two stores coincide; cross-store
clones cannot find the source.
Proposed fix
Thread the source dataset's object store into the commit path and use it to read
the source manifest (
CommitBuilder::with_source_store), while the new dataset iswritten through the destination store. Stream files source → target when the
stores differ; keep the server-side copy fast-path when they match. PR to follow.
Notes
deep_cloneto the Python API): this is thecore correctness fix that lets such a clone cross accounts at all.
base_pathsremain out ofscope (Feature Request: Support Per-Base Independent Credentials in Multi-Base Datasets #6093).