fix(parquet): enable Zstandard-compressed reads - #243
Conversation
Enable the parquet crate's zstd feature so ExArrow can read Zstandard-compressed Parquet files. Previously, reads failed at runtime because default Parquet features were disabled and zstd was not explicitly enabled. Add an externally generated Zstandard fixture and regression test, and document compression support in the Parquet reader guide and changelog.
|
hey @thanos, thanks for this interesting package! Would you mind looking into this PR and say if it is something you would consider accepting? If you for some reasons don't want this, it's fine. Close this PR and then i'll maintain my own fork. Best, |
|
Sorry I missed your PR. I'm going to look at it tonight. Many thanks. |
|
Sorry again for the delay — yes, this is absolutely something we want. You’re right that zstd was broken on 0.7.2 because parquet features were too narrow. I'm about to release 0.8 - due to other work I've been sitting on it since 7th - with zstd (and snap/lz4/gzip) enabled for both read and write; I’ll fold your DuckDB fixture and regression test into that and credit you in the changelog. No need to maintain a fork for this. |
|
Thanks so much for this, @mindreframer — really appreciate you catching the Zstd gap and shipping a clean fix with a third-party fixture. Closing this PR in favor of the v0.8.0 work on #244, which already enables Zstd (and the other Parquet codecs) in the native crate. We folded your DuckDB-generated parquet_zstd.parquet fixture and the external-read regression into that branch so we keep the interop coverage you added — credited in the CHANGELOG and test/fixtures/README.md. Your PR was the right diagnosis and a very helpful contribution. Looking forward to more from you if you’re up for it. — Thanos |
|
@thanos Thanks, looking forward to play with it! I like your selection of packages on hex.pm, some are really esoteric :) |
|
BTW, because I was not sure about the status of this issue, i've created a package with a narrower scope:
https://github.com/mindreframer/parquex maybe of interest to you, too. |
2 participants
Enable the parquet crate's zstd feature so ExArrow can read Zstandard-compressed Parquet files. Previously, reads failed at runtime because default Parquet features were disabled and zstd was not explicitly enabled.
Add an externally generated Zstandard fixture and regression test, and document compression support in the Parquet reader guide and changelog.