Bug Fix: Cast UINT32 to INT32 to ensure compatibility with other engines

[JeroenSchmidt]

Rationale for this change

• PyIceberg was preserving original Arrow types in Parquet files, causing Spark to fail with Unsupported logical type: UINT_32
• Extends the integer casting logic in ArrowProjectionVisitor._cast_if_needed to handle unsigned-to-signed conversions at the same bit width. Specifically uint32 -> int32.

Context:

This is a follow-up to #2799 (which fixed #2791) where uint8/uint16 casting was addressed. That fix only covered widening conversions (source_width < target_width), which missed the uint32 case since both uint32 and int32 are 32-bit. Without this cast, Parquet files are written with the UINT_32 physical type while Iceberg metadata declares INT_32, causing Spark to fail on read.

Changes / Are these changes tested?

pyiceberg/io/pyarrow.py: Extended the cast condition to also trigger when the source is an unsigned integer with the same (or smaller) bit width as the signed target
tests/io/test_pyarrow.py: Added (pa.uint32(), IntegerType(), pa.int32()) test case

Notes

• The cast uses PyArrow's default safe=True, so values exceeding INT32_MAX (2^31-1) will raise rather than silently corrupt
• Existing behavior for all other integer type combinations is unchanged

[sungwy]

Hi @JeroenSchmidt - thanks for contributing this PR! This seems consistent with #2799’s precedent of normalizing Arrow integer input to Iceberg’s canonical signed physical type for write compatibility.

On the edge case you highlighted: could we add a regression test showing that uint32 values > INT32_MAX fail rather than wrap/truncate?

[JeroenSchmidt]

Hi @JeroenSchmidt - thanks for contributing this PR! This seems consistent with #2799’s precedent of normalizing Arrow integer input to Iceberg’s canonical signed physical type for write compatibility.

On the edge case you highlighted: could we add a regression test showing that uint32 values > INT32_MAX fail rather than wrap/truncate?

Thank you @sungwy for having a look.
I pushed a test that ensures that we get an exception instead of writing wrapped data.

[JeroenSchmidt]

Hi @sungwy have you have a chance to have a second look?

[sungwy]

Thanks again for digging into this @JeroenSchmidt.

I thought about this a bit more, and taking a step back: the root issue is that we're writing non-conformant Parquet.

Table Spec: Appendix A, which should be our source of truth, maps int/long to signed int32/int64 with no unsigned annotation, so the UINT_32 we emit today as a result of Arrow→Iceberg conversion is already out of spec. This PR fixes that symptom, but I think the cleaner fix is one step earlier.

There are really two steps on the write path: (1) the Arrow→Iceberg mapping picks the Iceberg type for a uint32 column, then (2) the write casts the data to match it. Today step 1 (_ConvertToIceberg.primitive) keys only on bit width, so uint32 maps to int (int32). IMHO that's where the loss is locked in, since int32 can't hold uint32 values ≥ 2^31. This PR's same-width cast is then left narrowing into that a type that's already too small.

If step 1 instead mapped uint32 → long, step 2 becomes a plain lossless widening (uint32 → int64), it's spec-conformant, and the same-width cast isn't needed at all. Same idea for uint8/uint16 → int (already lossless).

The only cases with no lossless target are uint64 (no Iceberg type wider than int64) and a uint32 appended to an already-declared int column. There I'd rather reject the write with a clear error than silently cast it to its signed counterpart. A lossy conversion should be the caller's explicit choice, not something we do quietly.

So I'm hesitant to land the same-width cast as-is. Could we instead (1) map uint32 → long and (2) reject unsigned writes with no lossless target?