Repository navigation
[GH-3467] GeoPandas: implement read_feather - #3483
Conversation
| "It should only be used if the file is expected to be small." | ||
| ) | ||
| gpd_df = gpd.read_feather(path, columns=columns, **kwargs) | ||
| return GeoDataFrame(gpd_df) |
There was a problem hiding this comment.
[P2] Reuse the safe local GeoPandas conversion
Please use GeoDataFrame._from_local_geopandas(gpd_df) here, following the existing from_dict and from_features pattern (introduced in #3289).
On supported Spark 3.5.0, a valid Feather file with geometry=[None, Point(1, 2)] raises pyarrow.ArrowInvalid at this call: pandas-on-Spark checks the first object value when inferring the geometry UDT. The existing helper handles this leading-null case and preserves the data, CRS, and index.
I reproduced this locally on Spark 3.5.0 / GeoPandas 1.1.3: the leading-null read failed, while the trailing-null read and both helper controls passed (1 failed, 3 passed). The same input works on Spark 3.5.7, so the new tests passing there does not cover this compatibility case. Please add a regression test with the null geometry first.
There was a problem hiding this comment.
Hi Jia,
Thanks for catching the error and for the repro. I have switched to GeoDataFrame._from_local_geopandas(gpd_df) like from_dict/from_features, and added test_read_feather_leading_null_geometry to test the same. It has been tested on Spark 3.5.0: the new test fails with ArrowInvalid before the change and passes after.
Add sedona.spark.geopandas.read_feather, the reading counterpart of GeoDataFrame.to_feather (apache#3455). Feather is a single-file format with no Spark reader, so the file is read on the driver with GeoPandas and the result is distributed, the same approach to_feather takes for writing. The GeoPandas signature is kept: columns, to_pandas_kwargs and any PyArrow read options are passed through, and GeoPandas's handling of the geo metadata keeps the active geometry column, CRS and index. to_pandas_kwargs raises NotImplementedError on GeoPandas versions whose read_feather does not accept it, rather than failing with a TypeError. Also update the GeoPandas coverage reference. Its summary still counted to_feather as unsupported, so the totals now include both. Generated-by: Claude Code (Claude Opus 5.5)
Build the result with GeoDataFrame._from_local_geopandas, following from_dict and from_features. On Spark 3.5.0, pandas-on-Spark infers the geometry type from the first value only, so a Feather file whose first geometry is null raised pyarrow.ArrowInvalid. Add a regression test with the null geometry first. Generated-by: Claude Code (Claude Opus 5.5)
1cbf8ef to
f6f0899
Compare
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes GeoPandas: implement read_feather #3467What changes were proposed in this PR?
Adds
sedona.spark.geopandas.read_feather, the reading side ofGeoDataFrame.to_feather(#3455). Feather has no Spark reader, so the file is read on the driver with GeoPandas and then distributed, same asto_featherdoes for writing.It keeps the GeoPandas signature:
columns,to_pandas_kwargsand any PyArrow read options are passed through, and the active geometry column, CRS and index come back as they were written. On GeoPandas versions whoseread_featherdoesn't acceptto_pandas_kwargs, it raisesNotImplementedErrorinstead of a confusingTypeError.Also updates the GeoPandas coverage page, which still listed
to_featheras unsupported.How was this patch tested?
New tests in
python/tests/geopandas/test_io.py: round trips throughto_feather, column selection, CRS and index preservation, and theto_pandas_kwargspath. Ran on Python 3.8, 3.9 and 3.11, plus the fulltests/geopandassuite.Did this PR include necessary documentation updates?
Assisted by Claude Code