IO: Extract PyArrowFileIO into dedicated module

[qzyu999]

Summary

Extract PyArrowFile, PyArrowFileIO, PyArrowLocalFileSystem and their helper functions into a new pyiceberg/io/_pyarrow_fileio.py module. The original pyiceberg/io/pyarrow.py re-exports all public names for full backward compatibility.

This is the first step toward decomposing the 3,100+ line pyarrow.py monolith into focused modules by concern (tracked in #3737).

Changes

• New file: pyiceberg/io/_pyarrow_fileio.py (584 lines) containing the FileIO implementation
• Modified: pyiceberg/io/pyarrow.py (-537 lines) replaced inline definitions with re-exports
• Removed unused imports from pyarrow.py that were only needed by the FileIO section

What stays the same

• All existing from pyiceberg.io.pyarrow import PyArrowFileIO imports continue to work
• Zero behavior change pure code movement with re-exports
• Pickle/unpickle of PyArrowFileIO works unchanged
• No new dependencies

Why

pyarrow.py handles 6 unrelated concerns in one file. The FileIO concern (filesystem abstraction for S3/GCS/Azure/HDFS/local) has zero coupling to schema conversion, expression translation, or statistics collection. Extracting it:

1. Reduces cognitive load when working on scan/write logic
2. Establishes the pattern for subsequent extractions (schema, expressions, stats, write, scan)
3. Creates a clean seam point for future alternative FileIO implementations

Testing

Full unit test suite passes: 3803 passed, 3 skipped, 0 failures, 0 errors

(Windows local testing requires #3721 + #3722 applied on top; Linux CI should pass without them.)

[abnobdoss]

Given that #3737 plans several PyArrow-specific extractions, should we first convert pyiceberg/io/pyarrow.py into a package with a pure rename to pyiceberg/io/pyarrow/__init__.py, then extract concerns into submodules such as pyarrow/fileio.py?

[Fokko]

Is this really needed? I'm not sure I find this an improvement, since I'd now have to go back and forth between two highly coupled files.

[qzyu999]

Given that #3737 plans several PyArrow-specific extractions, should we first convert pyiceberg/io/pyarrow.py into a package with a pure rename to pyiceberg/io/pyarrow/__init__.py, then extract concerns into submodules such as pyarrow/fileio.py?

Hi @abnobdoss, that's a good idea, a pyarrow/ package with submodules (fileio.py, schema.py, etc.) is cleaner than underscore-prefixed flat files. The __init__.py would handle re-exports and the import path from pyiceberg.io.pyarrow import PyArrowFileIO stays stable.

Is this really needed? I'm not sure I find this an improvement, since I'd now have to go back and forth between two highly coupled files.

Hi @Fokko, I agree that on its own, this extraction doesn't add much, _pyarrow_fileio.py and pyarrow.py are still coupled through the re-exports. The value comes from the end state: once all six concerns are extracted, the remaining pyarrow.py is just a re-export shim, and each concern (FileIO, schema, scan, write, etc.) becomes independently testable and substitutable. The immediate goal is enabling a ComputeEngine protocol at the scan/write boundary so DataFusion can slot in for bounded-memory operations (e.g., #271, #1210, #1092) without touching a 3K-line file.

Would a single PR that converts pyarrow.py into a package (as @abnobdoss suggests) and splits all concerns at once be more palatable? This current direction is based on conversations (recently with @kevinjqliu, @rambleraptor, and others previously) during the past two Python sync calls. It started with #3554, then #3715 / #3716 which has been superseded by #3737 and this PR.