You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PooledChatModel leaks API-key slots on client construction failures and cancellation #760
On main at b6f0304, contrib/batch_scan/api_pool.py reserves a pool slot before building ChatOpenAI, but construction is outside the cleanup try/except. A client-construction exception leaves the slot occupied. Repeating this can exhaust the pool and leave subsequent batch calls waiting for capacity that will never return.
A local reproduction needs no live request or real key:
importosfromcontrib.batch_scan.api_poolimportApiKey, ApiKeyPool, PooledChatModelos.environ["SSL_CERT_FILE"] ="/missing/ca-bundle.pem"pool=ApiKeyPool([ApiKey(key="test", base_url=None, model="test", max_concurrent=1)])
model=PooledChatModel(pool)
try:
model.invoke("test")
exceptExceptionasexc:
print(type(exc).__name__)
print(pool.active_requests) # actual: 1; expected: 0pool.acquire(timeout=0.05) # times out despite no request being in flight
Async cancellation has two related cleanup gaps: CancelledError bypasses except Exception during an in-flight call, and a blocking acquire delegated to a thread can acquire a slot after its awaiting coroutine is cancelled.
I can wrap client construction and invocation in a finally-based release, and make async capacity waiting cancellable without an orphaned acquiring thread. Tests can cover construction failures, in-flight cancellation, waiting cancellation, normal capacity recovery and existing rate-limit retry behavior. This concerns per-call cleanup; #743 addresses owner/process lifetime cleanup separately.
Resolved by merged PR #796 (merge commit 7ee43713), verified on main b07f181d on 2026-10-13.
Client construction and invocation now share a finally-based slot release in both synchronous and asynchronous paths. Async capacity waiting uses cancellable polling instead of an orphanable acquiring thread.
Fresh offline checks reproduced the original invalid-CA constructor failure and verified active_requests == 0 plus successful capacity reuse. In-flight cancellation and cancellation while waiting also returned capacity correctly without a late orphan acquisition. Focused constructor, deadline, retry, and pool tests passed; no live provider requests were made.
Closing as completed. PR #763 remains open, but this issue's reported cleanup gaps were independently fixed by #796.
Activity
rng1995 commentedon Oct 14, 2026
Resolved by merged PR #796 (merge commit
7ee43713), verified on mainb07f181don 2026-10-13.Client construction and invocation now share a finally-based slot release in both synchronous and asynchronous paths. Async capacity waiting uses cancellable polling instead of an orphanable acquiring thread.
Fresh offline checks reproduced the original invalid-CA constructor failure and verified
active_requests == 0plus successful capacity reuse. In-flight cancellation and cancellation while waiting also returned capacity correctly without a late orphan acquisition. Focused constructor, deadline, retry, and pool tests passed; no live provider requests were made.Closing as completed. PR #763 remains open, but this issue's reported cleanup gaps were independently fixed by #796.