Parquet: Prune nested structs to the projected subtree

[jzhuge]

PruneColumns.struct() adds the original, unpruned Parquet field whenever the field's id is in selectedIds. A parent struct's id is in selectedIds whenever any of its descendants is projected, so projecting one leaf of a deeply nested struct widens the read schema back to the full intermediate struct.

This adds the pruned field instead. It also accumulates hasChange (|=), so a fully projected sibling visited later cannot reset pruning recorded by an earlier sibling at the same level.

Example: for l1 struct<l2 struct<x, l3 struct<leaf, big>>>, projecting l1.l2.l3.leaf previously read all of l2. With this change it reads only l1.l2.l3.leaf. Whole-struct and top-level projections produce the same read schema as before. In our environment, reading a single deep leaf from a ~17 GB table took about 8x the scan task time of an id-only read, close to reading the whole parent struct.

Notes for reviewers:

• A selected struct is still read in full. Projections expand a selected struct to its full subtree, so field == originalField at every level and the old path is kept.
• list() and map() are unchanged. Structs inside list elements and map values prune correctly once struct() returns the pruned field.
• The else if (field != null) branch now adds the pruned field instead of the original. This only matters for files where some fields have no ids. It should be more correct, but this is the area where review would help most.

This revives #12634 and #14744, which were closed as stale, and addresses #11332.

Tests: 6 new tests in TestPruneColumns. 4 fail on main and pass with this change (testDeeplyNestedStructProjection, testDeeplyNestedStructMixed, testDeeplyNestedStructInsideList, testDeeplyNestedStructPartiallyProjectedBeforeFullyProjected). 2 pin unchanged behavior (testDeeplyNestedStructWhole, testNestedStructInsideMap). ./gradlew :iceberg-parquet:check passes.

---

AI Disclosure

• Model: Claude Opus 5.5
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Prune Parquet columns to the projected subtree for nested structs, with regression tests.

[bryanck]

Thanks @jzhuge , very nice find! I just have a few minor comments.

[uros-b]

+1, LGTM!

Rereview

[bryanck]

Can you follow the guidance in AGENTS.md for AI authored PRs? There is a disclosure block and some other rules.

(Also one note for other reviewers, typically adding so many tests for a small change is discouraged, but in this case I feel it is worthwhile, given the nature of the change.)

[bryanck]

This PR should address an unresolved concern raised in #12634 by @rdblue .

My thought about the concern is that if, say, a partial struct is passed into the struct prune method, then it is safe to assume the intent was a partial struct rather than the full struct. This is consistent with TypeUtil.project() for example (though different than TypeUtil.select()).

If that is deemed acceptable, then at the very least we should add a comment similar to the one for TypeUtil.project().

[FrancisGodinho]

lgtm after resolving the rest of the comments

[jzhuge]

AI disclosure updated per AGENTS.md.

On the #12634 concern: added javadoc on ParquetSchemaUtil.pruneColumns. A partial struct in the expected schema is treated as intentional projection, consistent with TypeUtil.project(). Structs that are explicitly selected keep their full subtree.