TINKERPOP-2774: Add native-async WebSocket driver for gremlin-python#3527
TINKERPOP-2774: Add native-async WebSocket driver for gremlin-python#3527masterhugo wants to merge 4 commits into
Conversation
Introduces five new classes alongside the existing sync WebSocket driver, enabling applications to use gremlin-python inside a running asyncio event loop without creating or blocking a thread-per-connection: * AsyncAiohttpWSTransport – pure-coroutine WebSocket transport (aiohttp) * AsyncGremlinServerWSProtocol – async protocol; offloads deserialization to the default executor so the event loop is never blocked by CPU work * AsyncResultSet – asyncio.Queue-backed result set with async iteration (async for) and coroutine one()/all() methods * AsyncConnection – async connection with lazy connect; background asyncio Task drains partial (206) responses without blocking the caller * AsyncClient – drop-in async counterpart to Client; uses asyncio.Queue for the connection pool and exposes async submit/submit_async/close; supports the async context-manager protocol 48 new unit tests cover all five classes using unittest.IsolatedAsyncioTestCase (stdlib, no additional test dependencies). No existing code is modified; all sync WebSocket and HTTP paths are unchanged. Assisted-by: Claude Code:claude-sonnet-4-6
…g for gremlin-python Extends the native-async WebSocket driver with the full Gremlin DSL layer: * async_graph_traversal.py — AsyncGraphTraversalSource, AsyncGraphTraversal (multi-inherits GraphTraversal + AsyncTraversal so all terminal steps are awaitable), AsyncRemoteStrategy, AsyncTraversalStrategies, AsyncRemoteTraversal, AsyncTransaction, and the async_traversal() factory. * async_driver_remote_connection.py — AsyncDriverRemoteConnection with async submit(), submit_async(), submit_stream(), session management, and always-rollback-on-close for session connections. * async_client.py / async_connection.py — retry on InvalidatedConnectionError (up to pool_size attempts), per-connection WebSocket keepalive pings (ping_interval), default_request_options / query_timeout, asyncio.shield on close, and auto-reconnect on write reset. * examples/async_connections.py, examples/async_traversals.py — runnable examples covering connection patterns, DSL usage, concurrent queries, streaming, and transactions. * tests — 157 new unit tests (242 total passing) covering the new classes. CHANGELOG updated for 3.8.2. Assisted-by: Claude Code:claude-sonnet-4-6
a108e2e to
19b9ca5
Compare
aiohttp's lowercase aliases (WSMsgType.close, .closed, .error, .text) are deprecated in favour of the canonical uppercase forms. Switch to CLOSE, CLOSED, ERROR, TEXT to avoid future breakage. Assisted-by: Claude Code:claude-sonnet-4-6
Fixes Apache RAT check failure — the file was created empty and missing the required license header. Assisted-by: Claude Code:claude-sonnet-4-6
|
Hi Hugo, thank you for the PR! An async Python driver is definitely something that's been asked for by the community, and I'm glad to see the contribution! As this is a very large body of work, we do need a community discussion on the TinkerPop devlist (dev@tinkerpop.apache.org) first. If you could open a DISCUSS post on the design and reference this PR, it would be great. Please also reference the open Jira ticket on this topic: https://issues.apache.org/jira/browse/TINKERPOP-2774. Thanks! Another note, given the plan to move completely to HTTP in TinkerPop 4.0.0 and beyond, if we were to accept a fully async WebSocket driver for a 3.x.x branch, we would also need a parallel async driver with HTTP protocol. Is that something you would be interested in contributing as well? |
|
Thanks for the feedback! I’ve started a DISCUSS thread on dev@ referencing TINKERPOP-2774 and this PR, and added the JIRA link to the PR description. |
|
Perfect and thank you! Note we are in the middle of doing some releases, so we might not get to reviewing this until next week. As well, since this is a big feature PR, given how the DISCUSS thread goes, we might take some time to finish the review. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 3.8-dev #3527 +/- ##
==========================================
Coverage ? 76.50%
Complexity ? 14913
==========================================
Files ? 1160
Lines ? 72219
Branches ? 8070
==========================================
Hits ? 55254
Misses ? 14023
Partials ? 2942 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Also, I've just enabled the GitHub actions for the PR, and it seems like there are some test failures. It might be something to look into in the meanwhile. |
JIRA: https://issues.apache.org/jira/browse/TINKERPOP-2774
Summary
Introduces a complete native-async driver for gremlin-python, enabling applications to use the full Gremlin DSL inside a running asyncio event loop without thread-pool wrappers or blocking calls.
Low-level transport layer (no existing code modified):
High-level DSL layer:
Examples: examples/async_connections.py and examples/async_traversals.py show connection patterns, DSL usage, concurrent queries with asyncio.gather, streaming, and transactions.
Motivation
Python async/await is now the standard concurrency model for web frameworks (FastAPI, Starlette, aiohttp). Applications running inside an event loop cannot use the existing sync Client without blocking the loop or pushing work to a thread. These classes fill that gap cleanly: the low-level layer handles transport and pooling; the DSL layer lets callers write await g.V().has_label("person").values("name").to_list() with no extra boilerplate.
Test plan
docker compose up --build --abort-on-container-exit gremlin-python-integration-tests
Checklist
Acknowledgements
Special thanks to Kuoun (Discord) for sharing the asyncio_gremlin reference implementation, which informed the robustness patterns in this driver (keepalive pings, retry logic, AsyncTransaction, and the async strategy pipeline).