Skip to content

PooledChatModel leaks API-key slots on client construction failures and cancellation #760

Closed

Description

@YMuskrat

On main at b6f0304, contrib/batch_scan/api_pool.py reserves a pool slot before building ChatOpenAI, but construction is outside the cleanup try/except. A client-construction exception leaves the slot occupied. Repeating this can exhaust the pool and leave subsequent batch calls waiting for capacity that will never return.

A local reproduction needs no live request or real key:

import os
from contrib.batch_scan.api_pool import ApiKey, ApiKeyPool, PooledChatModel

os.environ["SSL_CERT_FILE"] = "/missing/ca-bundle.pem"
pool = ApiKeyPool([ApiKey(key="test", base_url=None, model="test", max_concurrent=1)])
model = PooledChatModel(pool)
try:
    model.invoke("test")
except Exception as exc:
    print(type(exc).__name__)
print(pool.active_requests)  # actual: 1; expected: 0
pool.acquire(timeout=0.05)  # times out despite no request being in flight

Async cancellation has two related cleanup gaps: CancelledError bypasses except Exception during an in-flight call, and a blocking acquire delegated to a thread can acquire a slot after its awaiting coroutine is cancelled.

I can wrap client construction and invocation in a finally-based release, and make async capacity waiting cancellable without an orphaned acquiring thread. Tests can cover construction failures, in-flight cancellation, waiting cancellation, normal capacity recovery and existing rate-limit retry behavior. This concerns per-call cleanup; #743 addresses owner/process lifetime cleanup separately.

Activity

  1. rng1995 commented on Oct 14, 2026

    @rng1995
    Collaborator

    Resolved by merged PR #796 (merge commit 7ee43713), verified on main b07f181d on 2026-10-13.

    Client construction and invocation now share a finally-based slot release in both synchronous and asynchronous paths. Async capacity waiting uses cancellable polling instead of an orphanable acquiring thread.

    Fresh offline checks reproduced the original invalid-CA constructor failure and verified active_requests == 0 plus successful capacity reuse. In-flight cancellation and cancellation while waiting also returned capacity correctly without a late orphan acquisition. Focused constructor, deadline, retry, and pool tests passed; no live provider requests were made.

    Closing as completed. PR #763 remains open, but this issue's reported cleanup gaps were independently fixed by #796.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions