Skip to content

feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs#2866

Open
mbutrovich wants to merge 4 commits into
apache:mainfrom
mbutrovich:dv-codec
Open

feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs#2866
mbutrovich wants to merge 4 commits into
apache:mainfrom
mbutrovich:dv-codec

Conversation

@mbutrovich

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

Adds DeleteVector::deserialize, which decodes a deletion-vector-v1 Puffin blob into a DeleteVector (a RoaringTreemap of deleted row positions). This is the first, self-contained piece of deletion-vector read support (epic #2792): it adds no scan, reader, or planning changes and can be reviewed in isolation.

The blob layout follows the Iceberg Puffin spec and matches Iceberg-Java's BitmapPositionDeleteIndex / RoaringPositionBitmap:

[length: u32 big-endian][magic: D1 D3 39 64][vector][crc32: u32 big-endian]
  • length counts the magic and vector bytes (not itself or the CRC).
  • The CRC-32 (via crc32fast) covers the magic and vector.
  • vector is a roaring bitmap in the portable 64-bit format, read directly by RoaringTreemap::deserialize_from (roaring 0.11.x implements this official format).

Validation order is length prefix, then CRC (before decoding any payload), then magic, then the roaring decode, so a corrupt blob yields a single clear error rather than an opaque roaring failure. Cardinality is intentionally not checked here; the caller validates the decoded length against the delete file's record_count once the manifest metadata is available (a later PR in the epic).

Also adds crc32fast as a dependency.

This overlaps with #2414, which implemented the same decode. That PR has been idle since review feedback, so this is submitted in its place to keep the epic moving; thanks to @Shekharrajak for the original contribution and for informing this implementation.

Are these changes tested?

Yes, unit tests in delete_vector.rs:

  • Round-trip decode of empty, small, and 64-bit-key-spanning vectors.
  • A run-optimized bitmap (via RoaringTreemap::optimize) so the RUN-container decode path is exercised, since Iceberg-Java run-length-encodes deletion vectors before writing.
  • Error paths: too-short blob, length-prefix mismatch, bad magic, and bad CRC.

This decode path is also exercised end to end against deletion vectors written by Spark / Iceberg-Java in a draft DataFusion Comet PR (apache/datafusion-comet#4887), which reads real V3 merge-on-read tables through this code and verifies the deleted rows match Spark. That gives cross-implementation confidence that it decodes what Iceberg actually writes, not just what these tests encode.

Add DeleteVector::deserialize to parse a deletion-vector-v1 blob
([length][magic][roaring][crc32], portable 64-bit roaring) into a
DeleteVector. Verifies the length prefix, CRC-32, and magic before
decoding, and leaves cardinality validation to the caller that holds the
delete file's record_count.

Tests cover round-trips (including a run-optimized bitmap that exercises
the RUN-container decode path) and the length, magic, and CRC error paths.

Refs apache#2792.

@anoopj anoopj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks correct to me.

Comment thread crates/iceberg/src/delete_vector.rs Outdated
Comment thread crates/iceberg/src/delete_vector.rs
@mbutrovich

Copy link
Copy Markdown
Collaborator Author

The code looks correct to me.

Thanks for the quick review, @anoopj! I addressed the feedback in 23df07b.

@hsiang-c hsiang-c left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mbutrovich, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants