Skip to content

Core: Don't prune equality delete manifest entries by non-key column stats - #18338

Open
findinpath wants to merge 1 commit into
apache:mainfrom
findinpath:findinpath/equality-deletes
Open

findinpath wants to merge 1 commit into
apache:mainfrom
findinpath:findinpath/equality-deletes

Conversation

@findinpath

@findinpath findinpath commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

An equality delete's match condition depends only on its equality_ids
columns. Per spec, the file may legitimately carry additional columns of
the deleted row, but their stats describe values that play no part in the
match condition. ManifestReader's stats-based pruning evaluated the scan's
row filter against all of a delete file's column stats, so a predicate on
a non-key column could incorrectly prune a delete manifest entry and leave
a row that should have been deleted in the query result.

Only skip metrics evaluation for an equality delete file when the row
filter references a column outside its equality_ids; pruning using the
equality-key columns' own stats remains sound and is preserved.

Additional context

https://iceberg.apache.org/spec/#equality-delete-files

Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.

Is there a specific writer that does this (spark? flink? something else?)

Oracle GoldenGate

Issue found through trinodb/trino trinodb/trino#31399

@github-actions github-actions Bot added the core label Oct 1, 2026
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch 2 times, most recently from 2ec8a70 to adcf116 Compare October 1, 2026 11:53
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch 2 times, most recently from f2cc7ae to efa28c2 Compare October 1, 2026 13:29
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/test/java/org/apache/iceberg/DeleteFileIndexTestBase.java Outdated
@pvary

pvary commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

I'm not entirely sure how much effort do we want to put into fixing this, as there is no current use-case for this. Especially considering that we would like to get rid of the equality deletes in the long run.
If we fix this then we should think about fixing this on the commit path as well (newDelete().deleteFromRowFilter), and this is where I'm really getting unconvinced.

…stats

An equality delete's match condition depends only on its equality_ids
columns. Per spec, the file may legitimately carry additional columns of
the deleted row, but their stats describe values that play no part in the
match condition. InclusiveMetricsEvaluator evaluated the scan's row filter
against all of a delete file's column stats, so a predicate on a non-key
column could incorrectly prune a delete manifest entry and leave a row
that should have been deleted in the query result.

Narrow stats to a file's equality_ids columns before metrics evaluation
for equality delete files; pruning using the equality-key columns' own
stats remains sound and is preserved.
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch from efa28c2 to 0f942a8 Compare October 7, 2026 04:36
@github-actions github-actions Bot added the API label Oct 7, 2026
@findinpath

findinpath commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I'm not entirely sure how much effort do we want to put into fixing this

@pvary This represents a silent correctness issue.
We definitely should want to put effort into fixing such issues. 🙏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants