fix: include NULL in delete predicate for evolved partition fields in dynamic_partition_overwrite

[GayathriSrividya]

Closes #3148

Root cause

dynamic_partition_overwrite builds its delete predicate using the current partition spec only:

delete_filter = self._build_partition_predicate(
    partition_records=partitions_to_overwrite,
    spec=self.table_metadata.spec(),   # always current spec
    schema=self.table_metadata.schema(),
)

After a partition spec evolution (e.g. adding a region field), data files written under the older spec carry NULL for the new field — because region simply wasn't part of the schema at write time.

The predicate produced for the new partition {category=A, region=us} is:

category = 'A' AND region = 'us'

The _StrictMetricsEvaluator correctly sees that spec-0 files have region = NULL for every row, so region = 'us' can never be ROWS_MUST_MATCH. Those files are silently kept, leaving stale data behind.

Fix

Detect which fields in the current spec were absent from at least one historical spec ("evolved" fields). For those fields, extend the per-field clause to also accept NULL:

category = 'A' AND (region = 'us' OR region IS NULL)

This causes the metrics evaluator to flag pre-evolution files (all-null region) as ROWS_MUST_MATCH and delete them, while correctly preserving files in other non-overlapping partitions (e.g. region = 'eu').

Verification

Added two unit tests for _build_partition_predicate (with / without evolved_source_ids) and manually confirmed the repro from the issue now produces [999] instead of [1, 2, 999].

make lint  ✓
pytest tests/table/  ✓  (304 passed)

[Fokko]

@GayathriSrividya Thanks for opening this PR, but I think you've pulled in some unrelated commits here

[GayathriSrividya]

@GayathriSrividya Thanks for opening this PR, but I think you've pulled in some unrelated commits here

Thanks, @Fokko. Could you please point out which commits look unrelated? I’ll clean up the branch and keep this PR focused only on the #3148 fix. Or I will close this and open fresh PR

[Fokko]

@GayathriSrividya

The changes around making the list operations return an iterator:

[GayathriSrividya]

@GayathriSrividya

The changes around making the list operations return an iterator:

Okay, noted. Sorry for the extra commits.

I’ll make clean changes and open a PR from a different branch. Please give me some time.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[GayathriSrividya]

@GayathriSrividya

The changes around making the list operations return an iterator:

Updated this PR with a clean history and removed unrelated iterator commits.
Could you please take another look?