Skip to content

perf(parquet): prune row selection with OffsetIndex - #597

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix-parquet-selection-offset-index
Jul 23, 2026
Merged

perf(parquet): prune row selection with OffsetIndex#597
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix-parquet-selection-offset-index

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

A row selection without predicates did not load the Parquet OffsetIndex. Arrow-rs could filter rows, but still fetched unselected data pages, including pages for nested columns.

This is independent of #590: that PR prunes files, while this change prunes pages inside a retained Parquet file. It also improves selected-range reads introduced by #592.

Changes

  • Load ColumnIndex for predicate page pruning.
  • Load OffsetIndex for predicates or row selections.
  • Add a nested List regression test that verifies sparse selection reduces page I/O.

Tests

  • cargo fmt --all -- --check
  • Parquet module tests: 41 passed
  • paimon library tests: 1,839 passed, 1 ignored
  • clippy with warnings denied

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 5c17d26 into apache:main Jul 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants