Skip to content

fix(python): async custom_builtins work inside Jupyter / running event loop#1336

Merged
chaliy merged 4 commits intomainfrom
claude/test-bashkit-jupyter-DtLRt
Apr 17, 2026
Merged

fix(python): async custom_builtins work inside Jupyter / running event loop#1336
chaliy merged 4 commits intomainfrom
claude/test-bashkit-jupyter-DtLRt

Conversation

@chaliy
Copy link
Copy Markdown
Contributor

@chaliy chaliy commented Apr 17, 2026

What

When execute_sync() is called from inside a running asyncio event loop
(Jupyter / IPython), Python raises RuntimeError: Cannot run the event loop while another loop is running — even for a brand-new private loop created
inside the Rust binding. Async custom_builtins callbacks silently failed
with exit_code=1 and a "coroutine was never awaited" warning.

Why

execute_sync() drives the Tokio future on the calling thread via
rt.block_on(). When that thread is Jupyter's event-loop thread, Python's
asyncio thread-local state marks it as "already running" and refuses any
run_until_complete call — even on a different loop object.

How

PyPrivateAsyncLoop::run_awaitable now checks asyncio.get_running_loop()
before calling run_until_complete. If a loop is already running on the
thread, it falls back to a daemon background thread with its own fresh
event loop. The coroutine is driven via context.run(loop.run_until_complete, coro) so ContextVars from the calling context propagate correctly. The Python
helper is cached on first use to avoid repeated module compilation.

The await execute() path is unaffected — async callbacks already run on the
caller's loop via pyo3_async_runtimes task scheduling.

Tests added

  • tests/test_jupyter_compat.py — dedicated test file with two simulation
    patterns:

    • asyncio.run(cell()) — most explicit Jupyter analogue: boots a real event
      loop then calls execute_sync() from inside it (4 tests covering Bash,
      ScriptedTool, ContextVars, multiple sequential calls)
    • @pytest.mark.asyncio — exact Jupyter match via pytest's managed loop
      (5 tests: Bash, BashTool, ScriptedTool, ContextVars)
    • await execute() caller-loop verification (4 tests: Bash, BashTool,
      ContextVars)
  • examples/jupyter_basics.ipynb — official root-level example notebook
    (async/sync execution, persistent VFS, sync and async custom_builtins,
    ContextVar tracing, ScriptedTool multi-tool pipeline); verified executable
    end-to-end with nbconvert

CI

  • Added examples/*.ipynb to path triggers in python.yml
  • Added nbconvert + jupyter-client + ipykernel to the examples job
  • New "Run notebooks" step executes jupyter_basics.ipynb with a 120 s
    timeout; a cell error fails the job

Spec

specs/python-package.md updated with a custom_builtins async callback
section documenting the three dispatch paths and ContextVar propagation
semantics.

chaliy added 4 commits April 17, 2026 03:53
When execute_sync() is called from within a running asyncio event loop
(Jupyter/IPython), Python forbids run_until_complete on any additional
loop on the same thread — even a brand-new private one.

Fix: PyPrivateAsyncLoop detects a running loop via asyncio.get_running_loop()
and falls back to a daemon background thread that owns a fresh event loop.
The thread runs the coroutine inside the session's captured ContextVar
snapshot so ContextVars propagate correctly despite the thread switch.

The bg_thread_runner Python helper is cached on first use to avoid repeated
module compilation.

Adds regression tests (execute_sync works + ContextVar preserved inside a
running loop) and a Jupyter compatibility notebook exercise.
- Add tests/test_jupyter_compat.py covering all Jupyter-specific paths:
    asyncio.run() simulation (most explicit: boots a loop then calls
    execute_sync inside it), @pytest.mark.asyncio live-loop tests, and
    await execute() caller-loop tests. Surfaces covered: Bash, BashTool,
    ScriptedTool. Removes the two inline tests added during the fix commit
    from test_registered_tools.py — they live here now.

- Add examples/jupyter_basics.ipynb as the official root-level example:
    async/sync execution, persistent VFS, sync and async custom_builtins,
    ContextVar tracing, and ScriptedTool multi-tool pipeline. Verified
    executable end-to-end with nbconvert.

- Update .github/workflows/python.yml:
    Add examples/*.ipynb to path triggers; install nbconvert + jupyter-client
    + ipykernel in the examples job; add a "Run notebooks" step that executes
    jupyter_basics.ipynb with a 120 s timeout.
… CI section

Also auto-formats jupyter_async_test.ipynb per ruff.
@chaliy chaliy merged commit 2d628b6 into main Apr 17, 2026
22 checks passed
@chaliy chaliy deleted the claude/test-bashkit-jupyter-DtLRt branch April 17, 2026 04:16
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant