You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fda3f40
Browse filesBrowse the repository at this point in the historyBrowse files
refactor(crawlers)!: Enter and exit only what a crawler creates (#2445)
### Description
- A crawler now enters and exits only the objects it creates. A session
pool, browser pool, HTTP client, rendering type predictor or PydanticAI
extractor passed to it is the caller's to enter before `run()`, and
`run()` raises otherwise. The crawler no longer closes it after a run or
resets it on the next one. `Statistics` stay the exception: the crawler
enters them on every run, so crawlers can share an instance one after
another, but not at the same time.
- `PydanticAiDirectExtractor` becomes an async context manager, and
session pools, statistics, HTTP clients and the default rendering type
predictor can be entered again after a failed or cancelled entry or
exit. For details, see "Crawlers manage only what they create" in the
upgrade guide. The docs examples and project templates follow in a
separate PR.
### Issues
- Part of: #2301
- Follow-up: #2439
### Testing
- New unit and integration tests for passed and own objects of every
crawler, and for failed or cancelled entries and exits.
*✍️ Drafted by Claude Code*
Copy file name to clipboardExpand all lines: docs/06_upgrading/upgrading_to_v2.md
+39-1Lines changed: 39 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -493,6 +493,39 @@ class MyHttpClient(HttpClient):
493
493
494
494
`SitemapRequestLoader.fetch_next_request` skips a URL while its request is still in progress, for example when another sitemap lists it too. In v1, it returned a request for the URL again, and `get_total_count` counted the URL twice.
495
495
496
+
## Crawlers manage only what they create
497
+
498
+
A crawler now enters and exits only the objects it creates, the same as Crawlee JS. An object you pass is yours to enter and exit. The `Statistics` and the event managers are the exceptions, since the crawler enters them on every run. You can enter an event manager yourself as well. In v1, the crawler entered a passed object that wasn't active yet and exited it when its run finished, even while another crawler still used it.
499
+
500
+
What you do with an object you pass depends on its type:
501
+
502
+
-`SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
503
+
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
504
+
-`Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time.
When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.
522
+
523
+
`RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`.
524
+
525
+
`AdaptivePlaywrightCrawler.rendering_type_predictor` is now read-only. Pass your predictor to the constructor instead.
526
+
527
+
The private `_additional_context_managers` argument of `BasicCrawler` is removed. To have the crawler enter and exit an object on every run, a subclass builds the object with `self._own_or_borrow()`. A subclass that gives its parent class a default object, such as an HTTP client, builds it this way before it calls `super().__init__()`.
528
+
496
529
## Resetting statistics and the session pool
497
530
498
531
`Statistics.reset()` is now synchronous and resets only the in-memory statistics, the same as Crawlee JS. In v1, it also cleared the persisted record. To clear the record too, call the new `reset_store()` once the statistics are exited:
@@ -599,16 +632,21 @@ To keep the response cookies out of the sessions, pass `save_response_cookies=Fa
599
632
```python
600
633
# v1
601
634
crawler = ParselCrawler(use_session_pool=False)
635
+
await crawler.run(urls)
602
636
603
637
# v2
604
638
from crawlee.sessions import SessionCookies, SessionPool
0 commit comments