Skip to content

Commit fda3f40

Browse files
authored
refactor(crawlers)!: Enter and exit only what a crawler creates (#2445)
### Description - A crawler now enters and exits only the objects it creates. A session pool, browser pool, HTTP client, rendering type predictor or PydanticAI extractor passed to it is the caller's to enter before `run()`, and `run()` raises otherwise. The crawler no longer closes it after a run or resets it on the next one. `Statistics` stay the exception: the crawler enters them on every run, so crawlers can share an instance one after another, but not at the same time. - `PydanticAiDirectExtractor` becomes an async context manager, and session pools, statistics, HTTP clients and the default rendering type predictor can be entered again after a failed or cancelled entry or exit. For details, see "Crawlers manage only what they create" in the upgrade guide. The docs examples and project templates follow in a separate PR. ### Issues - Part of: #2301 - Follow-up: #2439 ### Testing - New unit and integration tests for passed and own objects of every crawler, and for failed or cancelled entries and exits. *✍️ Drafted by Claude Code*
1 parent 9ea3fcd commit fda3f40

29 files changed

Lines changed: 969 additions & 213 deletions

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,39 @@ class MyHttpClient(HttpClient):
493493

494494
`SitemapRequestLoader.fetch_next_request` skips a URL while its request is still in progress, for example when another sitemap lists it too. In v1, it returned a request for the URL again, and `get_total_count` counted the URL twice.
495495

496+
## Crawlers manage only what they create
497+
498+
A crawler now enters and exits only the objects it creates, the same as Crawlee JS. An object you pass is yours to enter and exit. The `Statistics` and the event managers are the exceptions, since the crawler enters them on every run. You can enter an event manager yourself as well. In v1, the crawler entered a passed object that wasn't active yet and exited it when its run finished, even while another crawler still used it.
499+
500+
What you do with an object you pass depends on its type:
501+
502+
- `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
503+
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
504+
- `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time.
505+
506+
For example:
507+
508+
```python
509+
# v1
510+
crawler = ParselCrawler(session_pool=SessionPool(max_pool_size=10))
511+
await crawler.run(urls)
512+
513+
# v2
514+
session_pool = SessionPool(max_pool_size=10)
515+
crawler = ParselCrawler(session_pool=session_pool)
516+
517+
async with session_pool:
518+
await crawler.run(urls)
519+
```
520+
521+
When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.
522+
523+
`RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`.
524+
525+
`AdaptivePlaywrightCrawler.rendering_type_predictor` is now read-only. Pass your predictor to the constructor instead.
526+
527+
The private `_additional_context_managers` argument of `BasicCrawler` is removed. To have the crawler enter and exit an object on every run, a subclass builds the object with `self._own_or_borrow()`. A subclass that gives its parent class a default object, such as an HTTP client, builds it this way before it calls `super().__init__()`.
528+
496529
## Resetting statistics and the session pool
497530

498531
`Statistics.reset()` is now synchronous and resets only the in-memory statistics, the same as Crawlee JS. In v1, it also cleared the persisted record. To clear the record too, call the new `reset_store()` once the statistics are exited:
@@ -599,16 +632,21 @@ To keep the response cookies out of the sessions, pass `save_response_cookies=Fa
599632
```python
600633
# v1
601634
crawler = ParselCrawler(use_session_pool=False)
635+
await crawler.run(urls)
602636

603637
# v2
604638
from crawlee.sessions import SessionCookies, SessionPool
605639

640+
session_pool = SessionPool(create_session_settings={'max_usage_count': 1})
606641
crawler = ParselCrawler(
607642
retry_on_blocked=False,
608643
save_response_cookies=False,
609-
session_pool=SessionPool(create_session_settings={'max_usage_count': 1}),
644+
session_pool=session_pool,
610645
)
611646

647+
async with session_pool:
648+
await crawler.run(urls)
649+
612650
# In a request handler
613651
response = await context.send_request(url, cookies=SessionCookies())
614652
```

‎src/crawlee/_utils/context.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,42 @@
33
import inspect
44
from collections.abc import Callable
55
from functools import wraps
6-
from typing import Any, TypeVar, cast
6+
from typing import TYPE_CHECKING, Any, Protocol, TypeVar, cast
7+
8+
if TYPE_CHECKING:
9+
from types import TracebackType
710

811
T = TypeVar('T', bound=Callable[..., Any])
912

1013

14+
class Collaborator(Protocol):
15+
"""An async context manager that reports through `active` whether it's entered."""
16+
17+
@property
18+
def active(self) -> bool:
19+
"""Whether the object is entered and not exited yet."""
20+
21+
async def __aenter__(self) -> Any:
22+
"""Prepare the object for use.
23+
24+
Raises:
25+
RuntimeError: If the object is already active.
26+
"""
27+
28+
async def __aexit__(
29+
self,
30+
exc_type: type[BaseException] | None,
31+
exc_value: BaseException | None,
32+
exc_traceback: TracebackType | None,
33+
/,
34+
) -> Any:
35+
"""Release what the object holds.
36+
37+
Raises:
38+
RuntimeError: If the object is not active.
39+
"""
40+
41+
1142
def ensure_context(method: T) -> T:
1243
"""Ensure the (async) context manager is initialized before executing the method.
1344

‎src/crawlee/autoscaling/_types.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from pydantic import Field
88
from pydantic.dataclasses import dataclass as pydantic_dataclass
99

10+
from crawlee._utils.context import Collaborator
1011
from crawlee._utils.docs import docs_group
1112

1213

@@ -104,12 +105,13 @@ class ConcurrencyConsumer:
104105

105106

106107
@docs_group('Autoscaling')
107-
class ConcurrencySystemInterface(Protocol):
108+
class ConcurrencySystemInterface(Collaborator, Protocol):
108109
"""The contract between an `AutoscaledPool` and the component that governs its concurrency.
109110
110111
The system answers whether there is free compute for one more task and tracks the budget that tasks are booked
111112
against. `ConcurrencySystem` is the default implementation. Anything matching this contract can take its place,
112-
without depending on its internals.
113+
without depending on its internals. An implementation is an async context manager, and `AutoscaledPool.run` refuses
114+
to run on one that isn't entered.
113115
114116
Every allocation method is told which `ConcurrencyConsumer` is asking, so an implementation may allocate per
115117
consumer. `ConcurrencySystem` does not, and serves whoever asks first, which can starve a consumer that joins
@@ -127,10 +129,6 @@ def desired_concurrency(self) -> int:
127129
def current_concurrency(self) -> int:
128130
"""The number of parallel tasks currently booked against this system."""
129131

130-
@property
131-
def active(self) -> bool:
132-
"""Whether the system is ready to be booked against. `AutoscaledPool.run` refuses to run when it is not."""
133-
134132
def has_capacity_for_task(self, consumer: ConcurrencyConsumer) -> bool:
135133
"""Report whether one more task may start right now on behalf of `consumer`.
136134

‎src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py‎

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,8 @@ def __init__(
135135
136136
Args:
137137
rendering_type_predictor: Object that implements RenderingTypePredictor and is capable of predicting which
138-
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used.
138+
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used. Keep a
139+
predictor passed in open with `async with` until every crawler using it has finished.
139140
result_checker: Function that evaluates whether crawling result is valid or not.
140141
result_comparator: Function that compares two crawling results and decides whether they are equivalent.
141142
static_parser: Implementation of `AbstractHttpParser`. Parser that will be used for static crawling.
@@ -144,11 +145,10 @@ def __init__(
144145
save_response_cookies: Whether to save the response cookies to the session, in both the static and the
145146
browser sub crawler.
146147
statistics: A custom `Statistics[AdaptivePlaywrightCrawlerStatisticState]` instance, allowing the use of
147-
non-default configuration.
148+
non-default configuration. The crawler enters it on every run, so don't enter it yourself.
148149
kwargs: Additional keyword arguments to pass to the underlying `BasicCrawler`.
149150
"""
150151
# Adaptive crawling related.
151-
self.rendering_type_predictor = rendering_type_predictor or DefaultRenderingTypePredictor()
152152
self.result_checker = result_checker or (lambda _: True)
153153
self.result_comparator = result_comparator or create_default_comparator(result_checker)
154154

@@ -164,6 +164,8 @@ def __init__(
164164

165165
super().__init__(statistics=adaptive_statistics, **kwargs)
166166

167+
self._rendering_type_predictor = self._own_or_borrow(rendering_type_predictor, DefaultRenderingTypePredictor)
168+
167169
# Sub crawlers related.
168170
playwright_crawler_specific_kwargs = playwright_crawler_specific_kwargs or _PlaywrightCrawlerAdditionalOptions()
169171

@@ -180,15 +182,16 @@ def __init__(
180182
# Initialize sub crawlers to create their pipelines.
181183
static_crawler_class = AbstractHttpCrawler.create_parsed_http_crawler_class(static_parser=static_parser)
182184

185+
# The sub crawlers never run themselves, so this crawler enters and exits their statistics.
183186
static_crawler = static_crawler_class(
184187
parser=static_parser,
185188
save_response_cookies=save_response_cookies,
186-
statistics=_NonPersistentStatistics(),
189+
statistics=self._own_or_borrow(None, _NonPersistentStatistics),
187190
**basic_crawler_kwargs_for_static_crawler,
188191
)
189192
playwright_crawler = PlaywrightCrawler(
190193
save_response_cookies=save_response_cookies,
191-
statistics=_NonPersistentStatistics(),
194+
statistics=self._own_or_borrow(None, _NonPersistentStatistics),
192195
**playwright_crawler_specific_kwargs,
193196
**basic_crawler_kwargs_for_pw_crawler,
194197
)
@@ -229,19 +232,22 @@ async def adaptive_post_navigation_hook_pw(context: PlaywrightPostNavCrawlingCon
229232
static_crawler.post_navigation_hook(adaptive_post_navigation_hook_static)
230233
playwright_crawler.post_navigation_hook(adaptive_post_navigation_hook_pw)
231234

232-
self._additional_context_managers = [
233-
*self._additional_context_managers,
234-
self.rendering_type_predictor,
235-
static_crawler.statistics,
236-
playwright_crawler.statistics,
237-
playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
238-
]
235+
# The default pool is the one the browser sub crawler built.
236+
self._own_or_borrow(
237+
playwright_crawler_specific_kwargs.get('browser_pool'),
238+
lambda: playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
239+
)
239240

240241
# Sub crawler pipeline related
241242
self._pw_context_pipeline = playwright_crawler._context_pipeline # noqa:SLF001 # Intentional access to private member.
242243
self._static_context_pipeline = static_crawler._context_pipeline # noqa:SLF001 # Intentional access to private member.
243244
self._static_parser = static_parser
244245

246+
@property
247+
def rendering_type_predictor(self) -> RenderingTypePredictor:
248+
"""The predictor that decides which sub crawler processes a request."""
249+
return self._rendering_type_predictor
250+
245251
@classmethod
246252
def with_beautifulsoup_static_parser(
247253
cls,

‎src/crawlee/crawlers/_adaptive_playwright/_rendering_type_predictor.py‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,11 @@ def __init__(self) -> None:
6868
# Flag to indicate the state.
6969
self._active = False
7070

71+
@property
72+
def active(self) -> bool:
73+
"""Whether the predictor is initialized and not cleared since."""
74+
return self._active
75+
7176
@abstractmethod
7277
def predict(self, request: Request) -> RenderingTypePrediction:
7378
"""Get `RenderingTypePrediction` based on the input request.
@@ -164,15 +169,20 @@ async def initialize(self) -> None:
164169
await super().initialize()
165170

166171
if not self._state.is_initialized:
167-
await self._state.initialize()
172+
try:
173+
await self._state.initialize()
174+
except BaseException:
175+
await super().clear()
176+
raise
168177

169178
@override
170179
async def clear(self) -> None:
171180
"""Clear the predictor state."""
172-
await super().clear()
173-
174-
if self._state.is_initialized:
175-
await self._state.teardown()
181+
try:
182+
if self._state.is_initialized:
183+
await self._state.teardown()
184+
finally:
185+
await super().clear()
176186

177187
@override
178188
def predict(self, request: Request) -> RenderingTypePrediction:

0 commit comments

Comments
 (0)