Skip to content

[python][daft] Support timestamp time travel in read_paimon#8527

Open
kerwin-zk wants to merge 1 commit into
apache:masterfrom
kerwin-zk:daft-timestamp-time-travel
Open

[python][daft] Support timestamp time travel in read_paimon#8527
kerwin-zk wants to merge 1 commit into
apache:masterfrom
kerwin-zk:daft-timestamp-time-travel

Conversation

@kerwin-zk

@kerwin-zk kerwin-zk commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

The Daft connector's read_paimon / explain_paimon_scan previously only supported time travel by snapshot_id and tag_name. This adds a timestamp argument so users can time-travel to the latest snapshot at or before a given time — the third form Paimon already supports at the core level (scan.timestamp-millis / scan.timestamp). Only the Daft wrapper was missing; core is unchanged.

timestamp is dispatched by type:

  • int (epoch milliseconds) or datetime.datetime -> scan.timestamp-millis (a naive datetime is interpreted in the local timezone)
  • str (e.g. '2026-07-09 10:00:00') -> scan.timestamp

snapshot_id, tag_name and timestamp are mutually exclusive; setting more than one raises ValueError.

read_paimon("db.orders", opts, timestamp="2026-07-09 10:00:00")
read_paimon("db.orders", opts, timestamp=datetime(2026, 7, 9, 10))
read_paimon("db.orders", opts, timestamp=1751990400000)

Docs: docs/docs/pypaimon/daft.md time-travel section updated with the timestamp example and parameter description.

Tests

Added to tests/daft/daft_integration_test.py:

  • test_read_paimon_with_timestamp — end-to-end: with a cutoff between two commits, both the millis and datetime forms return only the first snapshot.
  • test_timestamp_scan_option_mapping — unit coverage of the type→option mapping (int / str / datetime; bool and float rejected).
  • test_read_paimon_rejects_multiple_time_travel — parametrized mutual-exclusion checks (replaces the old two-way check).

All tests/daft/ pass locally (163 passed, 1 skipped); flake8 clean under dev/cfg.ini.

@kerwin-zk kerwin-zk force-pushed the daft-timestamp-time-travel branch from 136f717 to 270751a Compare July 9, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant