Prepare v0.8.0: larger-than-memory Parquet (pushdown, write options, … - #244
Merged
Conversation
…metadata, multi-file). ExArrow now reads only the Parquet you need: column projection, row-group selection, and a small predicate AST with statistics pruning, so peak memory scales with selected row groups rather than the whole file. Enable parquet-rs compression codecs (snap/zstd/lz4/gzip) so compressed reads and writes work; previously default features were off and Zstd failed at runtime. - Bump mix.exs and native crate to 0.8.0; add CHANGELOG 0.8.0. - Parquet.Reader/Stream.from_parquet accept :columns, :row_groups, :filters; Parquet.Reader.read_stats/1 reports selected vs skipped row groups. - Writer.to_file/4 and to_binary/3 take :compression, :row_group_size, :dictionary (Elixir validation in ExArrow.Parquet.Opts before the NIF). - ExArrow.Parquet.Metadata.from_file/1 and from_binary/1 (footer only). - Stream.from_parquet_files/2 and from_parquet_dir/2: lazy per-file open with schema checking; Enum.take/2 does not open later files. - Public ExArrow.IPC.File.write/3 and ExArrow.RecordBatch.concat/1 (existing NIFs). - CI Mix.install optional-dep smoke job; optional apache/arrow-testing fixture suite; Parquet Livebook, guide rewrite, pushdown bench, README vs Hex arrow.
…metadata, multi-file). ExArrow now reads only the Parquet you need: column projection, row-group selection, and a small predicate AST with statistics pruning, so peak memory scales with selected row groups rather than the whole file. Enable parquet-rs compression codecs (snap/zstd/lz4/gzip) so compressed reads and writes work; previously default features were off and Zstd failed at runtime. - Bump mix.exs and native crate to 0.8.0; add CHANGELOG 0.8.0. - Parquet.Reader/Stream.from_parquet accept :columns, :row_groups, :filters; Parquet.Reader.read_stats/1 reports selected vs skipped row groups. - Writer.to_file/4 and to_binary/3 take :compression, :row_group_size, :dictionary (Elixir validation in ExArrow.Parquet.Opts before the NIF). - ExArrow.Parquet.Metadata.from_file/1 and from_binary/1 (footer only). - Stream.from_parquet_files/2 and from_parquet_dir/2: lazy per-file open with schema checking; Enum.take/2 does not open later files. - Public ExArrow.IPC.File.write/3 and ExArrow.RecordBatch.concat/1 (existing NIFs). - CI Mix.install optional-dep smoke job; optional apache/arrow-testing fixture suite; Parquet Livebook, guide rewrite, pushdown bench, README vs Hex arrow.
Keep the stream.ex multi_open_path fix that matches {:ok, sch} =
schema(opened), dropping the unreachable {:error, _} clause.
Co-authored-by: Roman Heinrich <roman.heinrich@gmail.com>
…ces); nested struct fixture + regression test - closed #245 F-002 — Int32/Float32 filter values range-checked at open via validate_filter_types - closed #246 F-003 — Livebook pins → ~> 0.8.0 - F-005 — Path validation before coercion - closed #248 F-009 — Stronger compression / row-group / filter / pruning assertions Should-fix + cheap follow-ups F-004 — script/check_livebooks.sh + quickstart smoke in CI - closed #247 F-006/F-011/F-017 — opts_test.exs + filter value types + zstd 1..22 - closed #249, closed #256, closed #262 F-007/F-008/F-010/F-018/F-025 — Stream.close/1, idempotent EOF nil, schema-mismatch test, defensive schema case, telemetry uses current path - closed #252, closed #253, closed #263, closed #269 F-012–F-016, F-019–F-024, F-026–F-028 — docs/date/README/bench label, zstd cast, Int32/Float32/Boolean pruning, metadata/empty-write tests, atoms macro, CHANGELOG nits
…he remaining Nice-to-fix/holes):
F-009 — Metadata now exposes encodings; snappy test asserts dictionary: false has no DICTIONARY encodings and true does
F-019 — Boolean stats pruning uses full false < true comparisons (not only :eq)
F-002 — Added Float32 non-representable filter error test
F-022 — Empty batch list pinned to {:ok, …} with num_rows == 0
F-026 — PR #244 body corrected; optional workflow_dispatch arrow-testing CI job (non-blocking); CHANGELOG clarified
Flight.Client.connect/3 dispatched to ClientMock because another async suite had set :flight_client_impl, then Mox raised UnexpectedCallError (no expectation). Fix: the stub tests now call ClientImpl / DatabaseImpl directly so they don’t depend on that env. Concurrent mock suites no longer interfere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Larger-than-memory Parquet for ExArrow v0.8.0: column projection, row-group selection, and a small predicate AST with statistics pruning, so peak memory scales with selected row groups rather than the whole file. Enable parquet-rs compression codecs (snap/zstd/lz4/gzip) so compressed reads and writes work; previously default features were off and Zstd failed at runtime.
:columns,:row_groups,:filters; Parquet.Reader.read_stats/1 reports selected vs skipped row groups.:compression,:row_group_size,:dictionary(Elixir validation in ExArrow.Parquet.Opts before the NIF).00_quickstart.exssmoke; optional apache/arrow-testing suite is local/opt-in (script/fetch_arrow_testing.sh+mix test --include arrow_testing, also viaworkflow_dispatchCI without gating PRs).Test plan
mix testgreen on CI matrixbash script/check_livebooks.shelixir livebook/00_quickstart.exs(NIF build)