Skip to content

TINKERPOP-2774: Add native-async WebSocket driver for gremlin-python#3527

Open
masterhugo wants to merge 4 commits into
apache:3.8-devfrom
masterhugo:feature/async-websocket-transport
Open

TINKERPOP-2774: Add native-async WebSocket driver for gremlin-python#3527
masterhugo wants to merge 4 commits into
apache:3.8-devfrom
masterhugo:feature/async-websocket-transport

Conversation

@masterhugo

@masterhugo masterhugo commented Jul 19, 2026

Copy link
Copy Markdown

JIRA: https://issues.apache.org/jira/browse/TINKERPOP-2774

Summary

Introduces a complete native-async driver for gremlin-python, enabling applications to use the full Gremlin DSL inside a running asyncio event loop without thread-pool wrappers or blocking calls.

Low-level transport layer (no existing code modified):

  • AsyncAiohttpWSTransport — pure-coroutine WebSocket transport (aiohttp)
  • AsyncGremlinServerWSProtocol — async protocol; offloads deserialization to the default executor so the event loop is never blocked by CPU work
  • AsyncResultSet — asyncio.Queue-backed result set with async iteration (async for) and coroutine one()/all() methods
  • AsyncConnection — async connection with lazy connect, WebSocket keepalive pings (background asyncio.Task), auto-reconnect on write reset, and asyncio.shield on close
  • AsyncClient — async counterpart to Client; uses asyncio.Queue for the connection pool, retry on InvalidatedConnectionError, per-client default_request_options / query_timeout, and async context-manager support

High-level DSL layer:

  • AsyncDriverRemoteConnection — drop-in async replacement for DriverRemoteConnection; exposes submit(), submit_async(), and submit_stream() as coroutines; supports server-side sessions and always rolls back open transactions on close()
  • AsyncGraphTraversalSource / AsyncGraphTraversal / async_traversal() — full Gremlin DSL in async mode; terminal steps to_list(), next(), to_set(), has_next(), and iterate() are all awaitable
  • AsyncTransaction — asyncio.Lock-protected transaction with begin(), commit(), rollback(), and async with support

Examples: examples/async_connections.py and examples/async_traversals.py show connection patterns, DSL usage, concurrent queries with asyncio.gather, streaming, and transactions.

Motivation

Python async/await is now the standard concurrency model for web frameworks (FastAPI, Starlette, aiohttp). Applications running inside an event loop cannot use the existing sync Client without blocking the loop or pushing work to a thread. These classes fill that gap cleanly: the low-level layer handles transport and pooling; the DSL layer lets callers write await g.V().has_label("person").values("name").to_list() with no extra boilerplate.

Test plan

  • 242 unit tests pass (pytest tests/unit/), including:
    • 48 tests for the transport/protocol/connection/client layer (test_async_websocket.py)
    • 109 tests for the DSL layer (test_async_graph_traversal.py)
    • 85 tests for AsyncDriverRemoteConnection (test_async_driver_remote_connection.py)
  • All tests use unittest.IsolatedAsyncioTestCase (stdlib — no additional test dependencies)
  • Integration tests via Docker Compose:
    docker compose up --build --abort-on-container-exit gremlin-python-integration-tests

Checklist

  • ASF license header added to all new files
  • CHANGELOG.asciidoc updated (3.8.2 section)
  • No existing public APIs, serialization formats, or network protocols modified
  • No new dependencies introduced (aiohttp is already a required dependency)

Acknowledgements

Special thanks to Kuoun (Discord) for sharing the asyncio_gremlin reference implementation, which informed the robustness patterns in this driver (keepalive pings, retry logic, AsyncTransaction, and the async strategy pipeline).

Introduces five new classes alongside the existing sync WebSocket driver,
enabling applications to use gremlin-python inside a running asyncio event
loop without creating or blocking a thread-per-connection:

* AsyncAiohttpWSTransport  – pure-coroutine WebSocket transport (aiohttp)
* AsyncGremlinServerWSProtocol – async protocol; offloads deserialization to
  the default executor so the event loop is never blocked by CPU work
* AsyncResultSet – asyncio.Queue-backed result set with async iteration
  (async for) and coroutine one()/all() methods
* AsyncConnection – async connection with lazy connect; background asyncio
  Task drains partial (206) responses without blocking the caller
* AsyncClient – drop-in async counterpart to Client; uses asyncio.Queue for
  the connection pool and exposes async submit/submit_async/close; supports
  the async context-manager protocol

48 new unit tests cover all five classes using unittest.IsolatedAsyncioTestCase
(stdlib, no additional test dependencies).

No existing code is modified; all sync WebSocket and HTTP paths are unchanged.

Assisted-by: Claude Code:claude-sonnet-4-6
…g for gremlin-python

Extends the native-async WebSocket driver with the full Gremlin DSL layer:

* async_graph_traversal.py — AsyncGraphTraversalSource, AsyncGraphTraversal
  (multi-inherits GraphTraversal + AsyncTraversal so all terminal steps are
  awaitable), AsyncRemoteStrategy, AsyncTraversalStrategies, AsyncRemoteTraversal,
  AsyncTransaction, and the async_traversal() factory.
* async_driver_remote_connection.py — AsyncDriverRemoteConnection with async
  submit(), submit_async(), submit_stream(), session management, and
  always-rollback-on-close for session connections.
* async_client.py / async_connection.py — retry on InvalidatedConnectionError
  (up to pool_size attempts), per-connection WebSocket keepalive pings
  (ping_interval), default_request_options / query_timeout, asyncio.shield on
  close, and auto-reconnect on write reset.
* examples/async_connections.py, examples/async_traversals.py — runnable
  examples covering connection patterns, DSL usage, concurrent queries,
  streaming, and transactions.
* tests — 157 new unit tests (242 total passing) covering the new classes.

CHANGELOG updated for 3.8.2.

Assisted-by: Claude Code:claude-sonnet-4-6
@masterhugo
masterhugo force-pushed the feature/async-websocket-transport branch from a108e2e to 19b9ca5 Compare July 19, 2026 02:46
aiohttp's lowercase aliases (WSMsgType.close, .closed, .error, .text)
are deprecated in favour of the canonical uppercase forms. Switch to
CLOSE, CLOSED, ERROR, TEXT to avoid future breakage.

Assisted-by: Claude Code:claude-sonnet-4-6
Fixes Apache RAT check failure — the file was created empty and missing
the required license header.

Assisted-by: Claude Code:claude-sonnet-4-6
@xiazcy

xiazcy commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Hi Hugo, thank you for the PR! An async Python driver is definitely something that's been asked for by the community, and I'm glad to see the contribution! As this is a very large body of work, we do need a community discussion on the TinkerPop devlist (dev@tinkerpop.apache.org) first. If you could open a DISCUSS post on the design and reference this PR, it would be great. Please also reference the open Jira ticket on this topic: https://issues.apache.org/jira/browse/TINKERPOP-2774. Thanks!

Another note, given the plan to move completely to HTTP in TinkerPop 4.0.0 and beyond, if we were to accept a fully async WebSocket driver for a 3.x.x branch, we would also need a parallel async driver with HTTP protocol. Is that something you would be interested in contributing as well?

@masterhugo masterhugo changed the title Add native-async WebSocket driver for gremlin-python TINKERPOP-2774: Add native-async WebSocket driver for gremlin-python Jul 20, 2026
@masterhugo

Copy link
Copy Markdown
Author

Thanks for the feedback! I’ve started a DISCUSS thread on dev@ referencing TINKERPOP-2774 and this PR, and added the JIRA link to the PR description.
Re: the HTTP driver — yes, I’d definitely be interested in contributing that as well. I’d like to get this WebSocket driver reviewed and validated first, and once it’s in good shape I’d be happy to continue with a parallel async driver over HTTP for the 4.0.0 direction.

@xiazcy

xiazcy commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Perfect and thank you! Note we are in the middle of doing some releases, so we might not get to reviewing this until next week. As well, since this is a big feature PR, given how the DISCUSS thread goes, we might take some time to finish the review.

@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (3.8-dev@85f52ef). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             3.8-dev    #3527   +/-   ##
==========================================
  Coverage           ?   76.50%           
  Complexity         ?    14913           
==========================================
  Files              ?     1160           
  Lines              ?    72219           
  Branches           ?     8070           
==========================================
  Hits               ?    55254           
  Misses             ?    14023           
  Partials           ?     2942           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiazcy

xiazcy commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Also, I've just enabled the GitHub actions for the PR, and it seems like there are some test failures. It might be something to look into in the meanwhile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants