Skip to content

Add Python APIs for PTD tensor dictionaries - #22921

Open
Kilbex wants to merge 4 commits into
pytorch:mainfrom
Kilbex:python-ptd-tensor-dicts
Open

Kilbex wants to merge 4 commits into
pytorch:mainfrom
Kilbex:python-ptd-tensor-dicts

Conversation

@Kilbex

@Kilbex Kilbex commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds public save_ptd and load_ptd Python helpers for saving and loading PTD tensor dictionaries without constructing DataPayload objects directly. The helpers validate supported CPU tensors, preserve contiguous and channels-last layouts, normalize other strided layouts while preserving shape and values, and reject malformed tensor entries.

Includes the review fixes from #21236: saved entries use the default FlatTensorConfig segment alignment for C++ mmap compatibility, optional torch dtypes are skipped when unavailable, and the API documentation explains layout normalization.

Replaces #21236 following the request to open a new PR. The original fork was deleted and recreated, so the original PR could not be reopened. This PR uses the updated Kilbex:python-ptd-tensor-dicts branch, including commit 788378f, so CI can run on the current changes.

Test plan

PYTHONPATH=.. python -m pytest extension/flat_tensor/test/test_serialize.py -q — 13 tests and 20 subtests passed.

lintrunner extension/flat_tensor/README.md extension/flat_tensor/serialize/BUCK extension/flat_tensor/serialize/__init__.py extension/flat_tensor/serialize/serialize.py extension/flat_tensor/test/test_serialize.py — no lint issues.

The earlier C++ smoke-test results cover FlatTensorDataMap with MmapDataLoader, 128-byte alignment, direct mapped-buffer aliases, mixed dtypes, and an empty tensor.

Implementation authored with Claude. Replacement PR prepared with Codex.

@pytorch-bot

pytorch-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22921

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 15 Awaiting Approval

As of commit b41c610 with merge base a00e2a6 (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 18, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@nil-is-all nil-is-all added the module: extension Issues related to code under extension/ label Sep 18, 2026
@nil-is-all

Copy link
Copy Markdown
Contributor

Hi @Kilbex, could you resolve branch conflicts and ping once ready for review? Thanks for putting up a new PR!

@Kilbex
Kilbex force-pushed the python-ptd-tensor-dicts branch from d90391c to b41c610 Compare September 19, 2026 03:14
@Kilbex

Kilbex commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

@nil-is-all The branch is updated with the latest main, and GitHub confirms there are no merge conflicts. The update is pushed and ready for review.

Validation: PYTHONPATH=.. python -m pytest extension/flat_tensor/test/test_serialize.py -q — 16 tests and 20 subtests passed. Lint passed for all five changed files. Remote checks have started on the updated commit.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: extension Issues related to code under extension/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants