fix(python): async custom_builtins work inside Jupyter / running event loop#1336
Merged
fix(python): async custom_builtins work inside Jupyter / running event loop#1336
Conversation
When execute_sync() is called from within a running asyncio event loop (Jupyter/IPython), Python forbids run_until_complete on any additional loop on the same thread — even a brand-new private one. Fix: PyPrivateAsyncLoop detects a running loop via asyncio.get_running_loop() and falls back to a daemon background thread that owns a fresh event loop. The thread runs the coroutine inside the session's captured ContextVar snapshot so ContextVars propagate correctly despite the thread switch. The bg_thread_runner Python helper is cached on first use to avoid repeated module compilation. Adds regression tests (execute_sync works + ContextVar preserved inside a running loop) and a Jupyter compatibility notebook exercise.
- Add tests/test_jupyter_compat.py covering all Jupyter-specific paths:
asyncio.run() simulation (most explicit: boots a loop then calls
execute_sync inside it), @pytest.mark.asyncio live-loop tests, and
await execute() caller-loop tests. Surfaces covered: Bash, BashTool,
ScriptedTool. Removes the two inline tests added during the fix commit
from test_registered_tools.py — they live here now.
- Add examples/jupyter_basics.ipynb as the official root-level example:
async/sync execution, persistent VFS, sync and async custom_builtins,
ContextVar tracing, and ScriptedTool multi-tool pipeline. Verified
executable end-to-end with nbconvert.
- Update .github/workflows/python.yml:
Add examples/*.ipynb to path triggers; install nbconvert + jupyter-client
+ ipykernel in the examples job; add a "Run notebooks" step that executes
jupyter_basics.ipynb with a 120 s timeout.
… CI section Also auto-formats jupyter_async_test.ipynb per ruff.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When
execute_sync()is called from inside a running asyncio event loop(Jupyter / IPython), Python raises
RuntimeError: Cannot run the event loop while another loop is running— even for a brand-new private loop createdinside the Rust binding. Async
custom_builtinscallbacks silently failedwith
exit_code=1and a "coroutine was never awaited" warning.Why
execute_sync()drives the Tokio future on the calling thread viart.block_on(). When that thread is Jupyter's event-loop thread, Python'sasyncio thread-local state marks it as "already running" and refuses any
run_until_completecall — even on a different loop object.How
PyPrivateAsyncLoop::run_awaitablenow checksasyncio.get_running_loop()before calling
run_until_complete. If a loop is already running on thethread, it falls back to a daemon background thread with its own fresh
event loop. The coroutine is driven via
context.run(loop.run_until_complete, coro)so ContextVars from the calling context propagate correctly. The Pythonhelper is cached on first use to avoid repeated module compilation.
The
await execute()path is unaffected — async callbacks already run on thecaller's loop via
pyo3_async_runtimestask scheduling.Tests added
tests/test_jupyter_compat.py— dedicated test file with two simulationpatterns:
asyncio.run(cell())— most explicit Jupyter analogue: boots a real eventloop then calls
execute_sync()from inside it (4 tests covering Bash,ScriptedTool, ContextVars, multiple sequential calls)
@pytest.mark.asyncio— exact Jupyter match via pytest's managed loop(5 tests: Bash, BashTool, ScriptedTool, ContextVars)
await execute()caller-loop verification (4 tests: Bash, BashTool,ContextVars)
examples/jupyter_basics.ipynb— official root-level example notebook(async/sync execution, persistent VFS, sync and async custom_builtins,
ContextVar tracing, ScriptedTool multi-tool pipeline); verified executable
end-to-end with nbconvert
CI
examples/*.ipynbto path triggers inpython.ymlnbconvert+jupyter-client+ipykernelto the examples jobjupyter_basics.ipynbwith a 120 stimeout; a cell error fails the job
Spec
specs/python-package.mdupdated with acustom_builtinsasync callbacksection documenting the three dispatch paths and ContextVar propagation
semantics.