Skip to content

deep_clone fails across object stores/accounts: source manifest read through destination store #7543

Description

@sezruby

Summary

Dataset::deep_clone cannot clone a dataset into a **different object store /****
**account (e.g. from one Azure storage account to another). The copy of the data
files works, but the commit fails because the Operation::Clone handler reads the
source manifest through the destination object store, which cannot see the
source.

Repro (conceptual)

// source dataset opened with account-A credentials
let mut src = Dataset::open("abfss://fs@accountA.dfs.core.windows.net/ds").await?;
// clone into account B, supplying account-B credentials as store_params
src.deep_clone("abfss://fs@accountB.dfs.core.windows.net/ds_copy", version, Some(accountB_params)).await?;

Observed

The commit step fails resolving/reading the source manifest, e.g.:

DatasetNotFound { path: "<source>/_versions/N.manifest", ... }

because do_commit_new_dataset (the Operation::Clone arm) calls
resolve_version_location / load_manifest and reads the index section through
object_store — the destination store — rather than the source store.

Root cause

deep_clone copies files source → target correctly, but the new manifest is built
during commit by reading the source manifest. That read uses the destination
store. Same-store clones work only because the two stores coincide; cross-store
clones cannot find the source.

Proposed fix

Thread the source dataset's object store into the commit path and use it to read
the source manifest (CommitBuilder::with_source_store), while the new dataset is
written through the destination store. Stream files source → target when the
stores differ; keep the server-side copy fast-path when they match. PR to follow.

Notes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions