feat: support multi-base tables in merge insert with target base routing#7610
Merged
Merged
Conversation
github-actions
Bot
added
A-python
BubbleCal
approved these changes
Jul 3, 2026
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Merge insert now works with multi-base datasets and can route new fragments across target bases round-robin like a normal write, via MergeInsertBuilder::target_bases / target_base_names_or_paths in Rust and MergeInsertBuilder.target_bases in Python. Column patch files and deletion files stay in primary storage. Also reject empty target base lists in validate_and_resolve_target_bases, which previously panicked.
Extend cleanup_data_fragments to delete files whose base_id matches a resolved target base via that base's object store, wire it through do_write_fragments, the legacy merge path, and add missing post-write failure cleanup in FullSchemaMergeInsertExec.
… midway A task failure in update_fragments previously returned early, leaving already-written new fragments (including ones routed to target bases) behind. Abort in-flight tasks and delete collected new fragments before surfacing the error.
jackye1995
force-pushed
the
merge-insert-multi-base
branch
from
2142dab to
b61ca17
Compare
Reserve base id 0 (PRIMARY_BASE_ID) to refer to the dataset's primary storage in target_bases, and resolve a name-or-path entry equal to the dataset URI the same way, so writes and merge inserts can round-robin across primary storage plus registered bases, e.g. target_bases([0, 1, 2]).
…mmit conflict A RetryableCommitConflict discards the attempt and re-executes it, so the attempt's data files are provably uncommitted; delete them (including files routed to target bases, which version cleanup never scans) before retrying. Other commit errors may be ambiguous about whether the manifest was written, so files are left alone there.
target_all_bases(include_primary) resolves at execution time to every base registered in the manifest, with the dataset's primary storage as the first rotation slot when included. Available on WriteParams (with_target_all_bases), MergeInsertBuilder, and in Python on write_dataset, write_fragments, and merge_insert (include_primary defaults to True). CREATE mode includes initial_bases in the rotation.
…dataset FragmentCreateBuilder skipped target base resolution when only target_all_bases was set, silently writing to primary storage. Also resolve conflict-retry cleanup against the refreshed per-attempt dataset instead of the executor's original one, so bases added between attempts still resolve, and add target_all_bases to the _write_fragments stubs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merge insert previously hard-coded
target_bases: Noneon all of its write paths.This PR makes merge insert work with multi-base datasets and adds target base routing:
MergeInsertBuilder::target_bases(Vec<u32>)andtarget_base_names_or_paths(Vec<String>)in Rust;
MergeInsertBuilder.target_bases(List[str])in Python, with the samename-or-path semantics as
write_dataset(target_bases=...).DataFile.base_idstamped, identically to a normal write, on all three write paths:the v2 plan (
FullSchemaMergeInsertExec), the legacy indexed full-schema path, andthe legacy partial-schema path.
in primary storage, consistent with
dataset.update()anddataset.delete().validate_and_resolve_target_bases(validation, name/path lookup, per-basecredentials).
target_baseskeeps the existing default: new files go to primary storage.Also fixes a pre-existing panic: an empty
target_baseslist hit remainder-by-zero inround-robin writer selection; it is now rejected with a descriptive error.
Tests cover merge insert on multi-base tables across all execution paths (with and
without routing), both base layouts (
is_dataset_roottrue/false), round-robindistribution across multiple files, mixed-base fragments from column patches, and
validation errors, in both Rust and Python.
PRIMARY_BASE_ID) — or an entry equal to the dataset URI in the namesvariant — includes the dataset's primary storage in the rotation, so
target_bases([0, 1, 2])spreads new files across primary plus bases 1 and 2. Appliesto normal writes, fragment writes, and merge insert.
target_all_bases(include_primary)convenience (WriteParams, MergeInsertBuilder,Python
write_dataset/write_fragments/merge builder;include_primarydefaults toTrue in Python): resolves at execution time to every registered base, primary first
when included; CREATE mode includes
initial_basesin the rotation.Follow-ups (out of scope)
target_basesyet — Java has no multi-base write supportat all today; parity should come with general Java multi-base write support.