[Kaizer (kaizerch)]

Overview

👉 TL;DR: Python connectors built on HttpStream can now turn on TCP keepalive and request timeouts by overriding a property, the same way they turn on use_cache. Both are off by default, so no existing connector changes.

#1168 added opt-in keepalive and timeouts to HttpRequester (low-code), but HttpStream builds its HttpClient without them, and HttpClient never modifies a session passed into it. So a Python connector can only get keepalive by mounting TcpKeepaliveHTTPAdapter on the client's private session, which is what source-salesforce does today (airbytehq/airbyte#91215).

Tracks https://github.com/airbytehq/airbyte-internal-issues/issues/18622 (part 1).

Changes

• HttpStream.use_tcp_keepalive (default False): when True, the stream's session mounts TcpKeepaliveHTTPAdapter, so a dropped connection fails in about two minutes instead of blocking a read forever.
• HttpStream.request_timeout (default None): a number or a (connect, read) pair sent with every request. A timeout returned by request_kwargs() still takes precedence.
• Both are passed into the HttpClient that HttpStream.__init__ builds. HttpClient already supported both parameters.

Review Spotlight

• http.py#L54-L69: the two arguments passed into HttpClient.
• http.py#L110-L124: the two properties.

Risks

• No existing connector changes behavior. Both defaults match what HttpClient used before. No connector in the airbyte monorepo defines a request_timeout or use_tcp_keepalive attribute that would now be picked up.
• The properties are read during HttpStream.__init__, so an override can't depend on attributes a subclass sets after calling super().__init__(). use_cache already works this way.

Post-merge actions

• Release as a minor version (7.48.0 if nothing else lands first).
• Open the source-salesforce follow-up against the release: set use_tcp_keepalive = True and remove its hand-mounted adapter (built and tested locally against this branch).

Test plan

• Four new tests in test_http.py: a default stream mounts the plain adapter and sends no timeout; an opted-in stream mounts TcpKeepaliveHTTPAdapter and sends (30, 180); a timeout from request_kwargs() wins. With the http.py change reverted, the two opted-in tests fail and the two others pass.
• unit_tests/sources/streams/http: 556 passed, 3 skipped.
• poetry run pytest unit_tests -m "not slow and not requires_creds": 6,454 passed, 39 skipped.
• ruff check, ruff format --check and mypy on http.py pass.
• source-salesforce on this branch with use_tcp_keepalive = True and its manual mount removed: its REST session still gets the keepalive adapter, and the full connector unit suite passes (SF_SUITE_RESULT).

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@kaizerch/feat/http-stream-tcp-keepalive-timeout#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch kaizerch/feat/http-stream-tcp-keepalive-timeout

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

6 493 tests  +4   6 457 ✅ +4   6m 23s ⏱️ +39s
    1 suites ±0      36 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit fde1579. ± Comparison against base commit 43ce5ea.

[github-actions]

PyTest Results (Full)

6 496 tests  +4   6 460 ✅ +4   9m 7s ⏱️ - 1m 46s
    1 suites ±0      36 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit fde1579. ± Comparison against base commit 43ce5ea.