migrate embedded CPython to 3.14 with optional free-threaded build#1200
Merged
Conversation
yokofly
force-pushed
the
backport/issue-11704-cpython-3.14-migration
branch
from
cb8c43a to
1bdfeea
Compare
…d build Backports the embedded CPython 3.14 migration onto proton develop: bumps contrib/cpython to 3.14.6 plus the cpython-cmake build glue, embedded-runtime sources (src/CPython/*), the docker server image, and packager support for --enable-python-free-threaded (cp314t). Notes: - keeps proton naming throughout (paths, VOLUME, ENV) - AsynchronousMetrics: new mimalloc / smaps memory metrics adapted to proton's plain-double AsynchronousMetricValue (no per-metric documentation strings) - adapts PythonStreamingSource to proton's Streaming::ISource(Block, bool, LoggerPtr, ProcessorID) constructor - docker/server: the old python3.10 package preinstall is dropped (it cannot survive the 3.14 runtime staging); see the Dockerfile TODO - GitHub Actions workflows are left unchanged; the cp314t CI matrix is a follow-up Requires the preceding Poco libexpat 2.8.1 bump: CPython 3.14's pyexpat needs the newer expat API to compile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the deferred cp314t CI matrix from 87af695. Builds the embedded CPython free-threaded on the two supported targets (Linux x86 + macOS ARM native) across proton_ci, nightly_test, manual_trigger_build, and release_build, keeping the packager --enable-python-free-threaded flag and the docker --build-arg ENABLE_PYTHON_FREE_THREADED=1 coupled so the embedded binary and the staged runtime never disagree (PythonInterpreterInfo's ABI guard rejects a mismatch). - proton_ci / nightly / manual / release Linux x86: FT packager flag + server image build-arg (nightly arm-gated, manual python_udf-toggle-gated) - release Build_Native_Darwin_arm64: -DENABLE_PYTHON_FREE_THREADED=ON - release python-Linux-x86_64.tar.gz asset: rebuilt to cp314t (deadsnakes python3.14-nogil) instead of the stale python:3.10; dropped the timeplus-neutrino bundle (installable on demand via SYSTEM INSTALL PYTHON PACKAGE; not a test dependency), kept ensurepip + truststore so pip ships - FT stays off where unsupported: Linux ARM (#6944), macOS x86, and the cross-compiled darwin-arm build - docker/server + proton_unit_test Dockerfile comments reconciled to match Known follow-up: the macOS python-Darwin-arm64.tar.gz is a pre-built S3 artifact still on python3.10 and must be regenerated as cp314t out of band to match the FT macOS arm native binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yokofly
force-pushed
the
backport/issue-11704-cpython-3.14-migration
branch
from
1bdfeea to
7c23cad
Compare
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c23cadb10
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The smoke_test_address_x64 (cp314t free-threaded ASan) run failed on exactly four suites. Fix or skip each, and add coverage for what free-threading actually introduces. All verified against the CI image (timeplus/proton_testing:testing-address-x64-<sha>). Fixes: - python_package_management/01_python_package_restapi: bump pinned pandas 2.3.2 -> 3.0.3. 2.3.2 has no cp314t wheel so pip source-builds and the smoke runtime has no C compiler; 3.0.3 ships a cp314t wheel (and stays FT-clean, GIL off). Updated both the install VERSION and the version assertion. - alert: the alert UDFs `import requests`, which used to be provided by the timeplus-neutrino bundle (removed in the cp314t migration). Install requests explicitly in the per-case setup (pure-Python, ships cp314t wheels). Skips (genuine cp314t gaps, tracked in #12128): - external_stream_python/1_python_external_stream_kafka: kafka-python needs the zstandard codec to decode the broker's zstd-compressed messages, but zstandard/lz4 ship no cp314t wheel and the runtime has no compiler. Unskip once zstandard ships cp314t wheels. - python_udf_venv/00_venv_auto_version: the embedded free-threaded interpreter cannot create a venv (venv.EnvBuilder.create fails [Errno 38] copying the base executable into .venv/bin/python3.14t); the case also asserted a stale 3.10 version string. New free-threading regression coverage (python_udf_basic): - 18_free_threaded_gil_disabled: asserts the embedded interpreter runs with the GIL disabled (sys._is_gil_enabled() is False). Every other Python UDF test passes equally on a GIL build, so without this a silent regression to cp314 would ship green (it already happened once during this migration). - 19_free_threaded_concurrency: runs a Python UDF under a parallel query (max_threads=8) and asserts both correctness of the aggregate under concurrent execution and that the UDF genuinely runs on more than one native OS thread. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…heckpoint graft, module cleanup) Resolves the Codex review findings on PR #1200. Not compiled locally (no configured build dir); verified by static review against OSS patterns (RemoteSource) + clang-format — CI is the compile/test gate. 1. Free-threaded cancel race (PythonStreamingSource): under cp314t the GIL guard no longer serializes Python access, so onCancel() reading py_iterator could race finishPython() resetting it (data race / use-after-free). Add py_obj_mutex guarding the py_iterator member *lifetime*: readers take a strong ref via PyObjectPtr::borrow() under the lock and release it before any Python call; finishPython() detaches the member under the lock. The mutex is never held across a Python call, so a blocked iterator stays interruptible. 2. Complete the checkpoint graft (PythonStreamingSource): the backport moved the source to stateful Streaming::ISource + setLastProcessedSN() but dropped the override bodies (enterprise offsets-only checkpoint mode #11753), leaving the default doCheckpoint()/doRecover()/doResetStartSN() that throw NOT_IMPLEMENTED on any checkpointed pipeline reading a Python external stream. Add the overrides mirroring RemoteSource: doCheckpoint persists lastProcessedSN and emits a barrier chunk, doRecover restores it via setLastCheckpointSN, doResetStartSN is a no-op (Python iterators cannot seek). Offsets-only / dummy checkpoint — does not make generators replayable. 3. Module cleanup borrowed keys (CPython/Utils.cpp): clearModuleDictLikeCPython snapshotted borrowed PyDict_Next keys; the SetItem(None) pass decrefs old values, which can run __del__ that mutates/frees a still-queued key. Hold owned refs (std::vector<PyObjectPtr> via PyObjectPtr::borrow). Deferred (tracked, #12128): ABI detection in PythonInterpreterInfo uses runtime sys._is_gil_enabled() instead of a build-ABI indicator (Py_GIL_DISABLED/SOABI/ abiflags), which over-rejects a cp314t interpreter run with PYTHON_GIL=1. Added a TODO note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Server_Python Addresses enterprise PR #11852 review (r3463188992): replace the inline `#ifdef Py_GIL_DISABLED` block computing `embedded_free_threaded` with the existing constexpr `cpython::GILGuard::buildSupportsFreeThreading()`, which encapsulates the same build-capability check — removing conditional compilation from the server init path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@codex review |
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7fe6bf62a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…-replayable source) Addresses PR #1200 review r3464751498 (P1). The previous offsets-only graft mirrored RemoteSource (a *seekable* source): doCheckpoint persisted lastProcessedSN and doRecover restored it via setLastCheckpointSN. But Streaming::ISource::recover() then runs doResetStartSN(recovered_sn+1) + setLastProcessedSN(recovered_sn) whenever recovered_sn >= 0 — and a Python generator cannot seek, so doResetStartSN is a no-op. The source would advertise a recovered offset while the generator actually restarts at position 0, replaying already-emitted rows into recovered downstream state (duplicate results / repeated side effects). Mirror the established non-replayable-source precedent (GenerateRandomSource) instead: doCheckpoint notifies the coordinator (IProcessor::checkpoint, no state persisted) and emits a barrier chunk; doRecover and doResetStartSN are no-ops. With nothing recovered, recover()'s recovered_sn stays -1 and the reposition path is skipped, so the source never claims an offset it cannot honor. The source stays stateful so checkpointed pipelines still align the checkpoint barrier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@codex review |
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e6daeb760e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Addresses PR #1200 review r3464871376 (P2). PythonInterpreterInfo derived `free_threaded` from `sys._is_gil_enabled()`, which is the runtime GIL mode, not the build ABI. A cp314t interpreter run with PYTHON_GIL=1 reports the GIL as enabled, so `free_threaded` came out false and the Py_GIL_DISABLED ABI guard wrongly rejected an ABI-compatible interpreter (skipping Python init); the inverse could let a cp314t interpreter look acceptable to a GIL build. Use the build flag `sysconfig.get_config_var("Py_GIL_DISABLED")` instead (re-adds the `import sysconfig` dropped during the migration). Verified against the cp314t image: free_threaded reports true both by default AND under PYTHON_GIL=1, where the old runtime signal returned false. Replaces the earlier TODO(#12128) deferral. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up review pass on the cp314t migration (PR #1200), addressing issues that the green amd64 Linux CI does not exercise. Correctness: - PythonStreamingSource / PythonTableTransform onCancel(): the PyThreadState_SetAsyncExc fallback routes a KeyboardInterrupt by thread id, which is only a valid query-ownership token while the GIL is held. Under free-threading (cp314t) GILGuard no longer serializes, so the worker can finish, clear its tid, be recycled by the pipeline thread pool, and start another query's Python between the load() and the inject — landing the interrupt on an unrelated query. Gate the fallback behind `if constexpr (!buildSupportsFreeThreading())`: GIL builds keep current behavior, FT builds compile it out and cancel via cancel_requested + the iterator cancel()/close() hooks. - PythonPackage::refreshImportState: clear the pending exception when PyObject_HasAttrStringWithError returns -1 before `continue`, so it does not leak into the next loop iteration (the loop-tail PyErr_Clear is skipped by the continue). - Utils getOrAddMainModule: guard PyUnicode_FromString("__builtins__") against allocation failure; a NULL key would deref NULL inside PyDict_Contains' hash rather than return -1. Docs: - refreshImportState: document the free-threading limitation (global import-state mutation is not serialized against concurrent UDF imports under cp314t); proper quiesce-on-install is a tracked follow-up. Artifacts: - Drop the point-in-time perf investigation logs src/CPython/perf/{README,REPORT_FT_vs_GIL}.md. - Move probe_scaling.sh to utils/ (next to ft_python_soak.sh) and fix its server-detection pgrep, which used the enterprise `timeplusd` process name instead of OSS `proton`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Build_Native_Darwin_arm64 job builds a free-threaded (cp314t) binary
but was pulling python-Darwin-arm64.tar.gz — the legacy python 3.10
runtime — so the macOS arm release shipped an ABI-mismatched interpreter
(embedded UDF init fails the PythonInterpreterInfo guard).
Mirror what proton-enterprise already does: read a SEPARATE, ABI-tagged
object python3.14t-Darwin-arm64.tar.gz (uploaded once, out of band) and
rename it to the legacy basename locally, so the release asset name and
downstream install scripts stay unchanged AND the legacy 3.10 object —
shared with 3.x maintenance branches — is never overwritten.
Follow-up (one-time, manual): build a macOS arm64 cp314t runtime
(python.org freethreaded / python-build-standalone / source --disable-gil)
in the bin/python3.14t + lib/python3.14t layout and upload it to
${S3_PATH}python3.14t-Darwin-arm64.tar.gz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under the free-threaded (cp314t) build, onCancel's PyThreadState_SetAsyncExc
fallback is compiled out (it routes by thread id, which under FT can race a
thread-pool recycle and misroute the KeyboardInterrupt onto an unrelated
query's recycled PyThreadState). A consequence of that deliberate safety gate
is that a CPU-bound Python iterator/generator that never yields back to the
engine and exposes no cancel()/close() hook is not force-interruptible under
FT; cooperative cancellation (cancel()/close() hooks, or checking the flag
between yields) remains the FT contract.
Two pre-existing tests still encoded the GIL-era expectation that async-exc
interrupts a non-cooperative `while: pass` loop and so hung/failed on the FT
build:
- PythonStreamingSourceCancelWithAsyncInterruptStopsCleanly kept an
unconditional EXPECT_TRUE(finished_after_cancel) even though it already
had a stop_for_test() cooperative escape hatch for FT.
- PythonTableTransformCancelDuringGeneratorIterationDoesNotEmitPartialChunk
spun in `while True: pass` with no escape hatch at all.
Gate the async-interrupt assertions on GILGuard::buildSupportsFreeThreading():
the GIL build keeps asserting force-interrupt works; the FT build drives a
cooperative stop flag so the generator raises, which still verifies the real
invariant (a cancellation that cuts the generator short must not emit the
partial chunk). Test-only change; no production behavior change.
Mirrors enterprise; verified there against a cp314t (py314t) runtime: all 18
tests in both files pass, including the 3 cancel tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prior commit asserted EXPECT_FALSE(finished_after_cancel) on the FT build in PythonStreamingSourceCancelWithAsyncInterruptStopsCleanly, on the assumption that a non-cooperative `while: pass` iterator can never be cancelled under FT. That is wrong: the source observes cancel_requested at each generate() loop boundary, so executor.cancel() CAN end the iterator if it lands before the iterator re-enters its blocking loop — a timing race. CI wins that race (finished_after_cancel == true, ~58ms) and the EXPECT_FALSE failed; an isolated local run loses it (~500ms). The value is therefore non-deterministic under FT. Only assert finished_after_cancel on the GIL build (where async-exc makes it deterministically true). Under FT, don't assert it: the stop_for_test() cooperative path, the clean-shutdown join, and the single-block check remain the invariants. Mirrors enterprise; verified there against a cp314t (py314t) runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backport the smoke-test fixes that landed in enterprise (merge 7bac678) but were missing from this OSS branch, so the OSS cp314t smoke matches enterprise: - pandas/67_df_keys: expect Index dtype 'str' (newer pandas on cp314t) instead of the stale 'object'. - scipy/15_stats_zipfian_cdf: skip — scipy.stats.zipfian.cdf returns version-dependent values on the cp314t scipy wheel (#12128). - venv/01_venv_auto_gen: skip — `import autogen` PEP 420 namespace package is not resolved by the embedded free-threaded interpreter (#12128). - basic/18_free_threaded_gil_disabled, basic/19_free_threaded_concurrency: also run the FT-invariant checks on the p3k1 (3-node) cluster, matching enterprise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…prise #11485
0041_python_udf_numpy/34_sum_int64_numpy_scalar returned the aggregate
result wrapped in a list (`return [self.sum]`), which proton's scalar
insert path cannot convert to int64 ("'list' object cannot be
interpreted as an integer", UDF_INTERNAL_ERROR 2533). This surfaced once
the python-udf-library smoke scope ran on cp314t.
Match enterprise's corrected form (#11485): seed `np.int64(0)`, merge with
`+=`, and return the bare numpy.int64 scalar. Verified in the CI testing
image (cp314t free-threaded, numpy 2.5.0): bounded query returns 15 and
the streaming query emits 15 then 21.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yokofly
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://py-free-threading.github.io/tracking/