Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: Add Symbol.asyncDispose to disposable services for DX (#4032)
Jan Buchara7b2f1c74b1d
diff --git a/docs/guides/remote_browser.mdx b/docs/guides/remote_browser.mdxindex f02d41be4..61e7a64b2 100644--- a/docs/guides/remote_browser.mdx+++ b/docs/guides/remote_browser.mdx@@ -58,6 +58,29 @@ For Playwright you can choose the protocol via the `remoteBrowser.connection.pro`remoteBrowser` builds a pool the crawler owns and tears down. To share one remote pool across multiple crawlers, construct a <ApiLink to="browser-pool/class/RemoteBrowserPool">`RemoteBrowserPool`</ApiLink> yourself and pass it as the `browserPool` option instead — a pool supplied that way is never destroyed by the crawler, so you control its lifecycle. Use `remoteBrowser` *or* `browserPool`, not both.+Declare it with `await using` and the browsers (and their remote sessions) are released once you are done with the pool:++```ts+import { PlaywrightPlugin, RemoteBrowserPool } from '@crawlee/browser-pool';+import { PlaywrightCrawler } from 'crawlee';+import playwright from 'playwright';++await using browserPool = new RemoteBrowserPool({+ browserPlugins: [new PlaywrightPlugin(playwright.chromium)],+ endpoint: 'wss://production-sfo.browserless.io?token=xxx',+ maxOpenBrowsers: 2,+});++await new PlaywrightCrawler({ browserPool, requestHandler: async () => { /* ... */ } }).run(['https://crawlee.dev']);+await new PlaywrightCrawler({ browserPool, requestHandler: async () => { /* ... */ } }).run(['https://apify.com']);+```++:::note Requires Node.js 24++The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="browser-pool/class/RemoteBrowserPool#destroy">`destroy()`</ApiLink> yourself instead — it is what the disposal hook calls anyway.++:::+## Limitations- **`headless` and `launchOptions` don't apply.** The remote service controls headless mode and browser flags; configure them on the service side.diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdxindex 5a73722ce..1ce2caba7 100644--- a/docs/guides/scaling_crawlers.mdx+++ b/docs/guides/scaling_crawlers.mdx@@ -62,6 +62,12 @@ With that warning aside, this is how we pass those options. One thing to watch:{ConcurrencySystemSource}</CodeBlock>+:::note Requires Node.js 24++The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="core/class/ConcurrencySystem#stop">`stop()`</ApiLink> yourself in a `finally` block instead — it is what the disposal hook calls anyway.++:::+:::tip Capping the combined concurrency of several crawlersInjecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.diff --git a/docs/guides/scaling_crawlers_concurrencySystem.ts b/docs/guides/scaling_crawlers_concurrencySystem.tsindex 76f1041dd..ba3184420 100644--- a/docs/guides/scaling_crawlers_concurrencySystem.ts+++ b/docs/guides/scaling_crawlers_concurrencySystem.ts@@ -1,7 +1,9 @@import { CheerioCrawler, ConcurrencySystem } from 'crawlee';-// Advanced scaling options live on a pre-configured ConcurrencySystem-const concurrencySystem = new ConcurrencySystem({+// Advanced scaling options live on a pre-configured ConcurrencySystem.+// An injected system's lifecycle is owned by us, not the crawler - `await using` stops it for us+// once we are done with it.+await using concurrencySystem = new ConcurrencySystem({// ...});@@ -10,10 +12,5 @@ const crawler = new CheerioCrawler({// ...});-// An injected system's lifecycle is owned by us, not the crawlerawait concurrencySystem.start();-try {- await crawler.run(['https://crawlee.dev']);-} finally {- await concurrencySystem.stop();-}+await crawler.run(['https://crawlee.dev']);diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdxindex 46cd82703..f019f13d0 100644--- a/docs/guides/session_management.mdx+++ b/docs/guides/session_management.mdx@@ -234,7 +234,23 @@ const listingCrawler = new CheerioCrawler({ sessionPool, requestHandler: async (const detailCrawler = new PlaywrightCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });```-A pool you construct yourself is owned by you, not the crawler — the crawler will never tear it down or reset it between runs. Call <ApiLink to="core/class/SessionPool#teardown">`teardown()`</ApiLink> when you are done with it to persist its final state and stop listening for persistence events.+A pool you construct yourself is owned by you, not the crawler — the crawler will never tear it down or reset it between runs. Dispose of it with `await using` when you are done, which persists its final state and stops it listening for persistence events:++```js+import { CheerioCrawler, SessionPool } from 'crawlee';++await using sessionPool = new SessionPool({ maxPoolSize: 100 });++const crawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });+await crawler.run(['https://crawlee.dev']);+// sessionPool is torn down here, even if the crawl throws+```++:::note Requires Node.js 24++The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink to="core/class/SessionPool#teardown">`teardown()`</ApiLink> yourself instead — it is what the disposal hook calls anyway.++:::## Custom session poolsdiff --git a/docs/public-api/crawlee-browser-pool.api.md b/docs/public-api/crawlee-browser-pool.api.mdindex 3f8863871..3692e6313 100644--- a/docs/public-api/crawlee-browser-pool.api.md+++ b/docs/public-api/crawlee-browser-pool.api.md@@ -170,6 +170,8 @@ export interface BrowserPluginOptions<LibraryOptions> {// @publicexport class BrowserPool<Options extends BrowserPoolOptions = BrowserPoolOptions, BrowserPlugins extends BrowserPlugin[] = InferBrowserPluginArray<Options['browserPlugins']>, BrowserControllerReturn extends BrowserController = ReturnType<BrowserPlugins[number]['createController']>, LaunchContextReturn extends LaunchContext = ReturnType<BrowserPlugins[number]['createLaunchContext']>, PageOptions = Parameters<BrowserControllerReturn['newPage']>[0], PageReturn extends UnwrapPromise<ReturnType<BrowserControllerReturn['newPage']>> = UnwrapPromise<ReturnType<BrowserControllerReturn['newPage']>>> extends TypedEmitter<BrowserPoolEvents<BrowserControllerReturn, PageReturn>> implements IBrowserPool<PageReturn> {+ // (undocumented)+ [Symbol.asyncDispose](): Promise<void>;constructor(options: Options & BrowserPoolHooks<BrowserControllerReturn, LaunchContextReturn, PageReturn>);// (undocumented)activeBrowserControllers: Set<BrowserControllerReturn>;@@ -568,6 +570,8 @@ export type RemoteBrowserEndpoint = string | ((options?: {// @publicexport class RemoteBrowserPool<Page = unknown> implements IBrowserPool<Page> {+ // (undocumented)+ [Symbol.asyncDispose](): Promise<void>;constructor(options: RemoteBrowserPoolOptions);readonly browserPool: BrowserPool;// (undocumented)diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.mdindex 24aba9fc6..3b24c8034 100644--- a/docs/public-api/crawlee-core.api.md+++ b/docs/public-api/crawlee-core.api.md@@ -160,6 +160,8 @@ export interface ConcurrencyConsumer {// @publicexport class ConcurrencySystem implements IConcurrencySystem {+ // (undocumented)+ [Symbol.asyncDispose](): Promise<void>;constructor(options?: ConcurrencySystemOptions);// (undocumented)get currentConcurrency(): number;@@ -1778,6 +1780,8 @@ export interface SessionOptions {// @publicexport class SessionPool implements ISessionPool {+ // (undocumented)+ [Symbol.asyncDispose](): Promise<void>;constructor(options?: SessionPoolOptions);addSession(options?: Session | SessionOptions): Promise<void>;getSession(sessionId?: string): Promise<Session | undefined>;diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.mdindex ea3f16af0..c18b6cdb5 100644--- a/docs/public-api/crawlee-playwright.api.md+++ b/docs/public-api/crawlee-playwright.api.md@@ -530,6 +530,8 @@ export type RenderingType = 'clientOnly' | 'static';// @publicexport class RenderingTypePredictor implements IRenderingTypePredictor {+ // (undocumented)+ [Symbol.asyncDispose](): Promise<void>;constructor(input: RenderingTypePredictorOptions);initialize(): Promise<void>;predict(input: Request_2): {@@ -537,6 +539,7 @@ export class RenderingTypePredictor implements IRenderingTypePredictor {detectionProbabilityRecommendation: number;};storeResult(requests: Request_2 | Request_2[], renderingType: RenderingType): void;+ teardown(): Promise<void>;}// Not exported by the entry point; reachable only as a referenced type.diff --git a/docs/tsconfig.json b/docs/tsconfig.jsonindex 68a912f0b..9df76ee8e 100644--- a/docs/tsconfig.json+++ b/docs/tsconfig.json@@ -2,7 +2,7 @@"extends": "../tsconfig.build.json","include": ["./**/*.ts"],"compilerOptions": {- "lib": ["ES2022", "DOM.AsyncIterable"],+ "lib": ["ES2022", "DOM.AsyncIterable", "ESNext.Disposable"],"noUnusedLocals": false}}diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex 8b02f15dc..bbe43ee88 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -17,7 +17,7 @@ This page summarizes the breaking changes in Crawlee v4. There are many, so the- **Timeouts that mean what they say.** Navigation and the request handler are [timed separately](#navigation-and-the-request-handler-are-timed-separately) — no more mysteriously summed limits — and a single route can get [its own timeout](#per-route-and-per-request-handler-timeouts) or extend it mid-flight.- **Composable crawling context.** The new `extendContext` option and `ContextPipeline` composition replace subclassing tricks for [adding members to the crawling context](#crawling-context-no-longer-includes-a-reference-to-the-crawler-itself).-- **Bring your own implementation.** Crawlers now accept any [`ISessionPool`](#custom-sessionpool-implementations-via-the-isessionpool-interface), [`IBrowserPool`](#custom-browserpool-implementations-via-the-ibrowserpool-interface), [`IRenderingTypePredictor`](#custom-rendering-type-predictors-via-the-irenderingtypepredictor-interface), [`IRequestManager`](#request-loaders-and-managers) or [`IStatistics`](#statisticsoptions-is-replaced-by-a-statistics-instance) — and never tear down an instance they did not create.+- **Bring your own implementation.** Crawlers now accept any [`ISessionPool`](#custom-sessionpool-implementations-via-the-isessionpool-interface), [`IBrowserPool`](#custom-browserpool-implementations-via-the-ibrowserpool-interface), [`IRenderingTypePredictor`](#custom-rendering-type-predictors-via-the-irenderingtypepredictor-interface), [`IRequestManager`](#request-loaders-and-managers) or [`IStatistics`](#statisticsoptions-is-replaced-by-a-statistics-instance) — and never tear down an instance they did not create, which [`await using` now does for you](#collaborators-you-own-are-disposable).- **One concurrency budget for several crawlers.** The new [`ConcurrencySystem`](#autoscaling-moved-to-concurrencysystem) can be shared between crawlers, capping their combined concurrency instead of letting each one oversubscribe the host.- **Native `fetch` types.** HTTP clients and `context.response` now use the [standard `Response`](#crawlingcontextresponse-is-now-of-type-response), and `got-scraping` is an [opt-in dependency](#http-client-packages-and-basehttpclient-reshaped) instead of a mandatory one.- **The session is the rotation unit.** A session carries its proxy, cookies and error score, and is rotated as a whole when blocked — replacing [proxy tiers](#tieredproxyurls-is-removed-from-proxyconfiguration) and [session rotation counters](#maxsessionrotations-and-requestsessionrotationcount-are-removed).@@ -71,6 +71,19 @@ Crawlee v4 is a native ESM package now. It can be still consumed from a CJS projSupport for older node versions was dropped.+### Collaborators you own are disposable++A crawler never tears down an instance it did not build, so anything you construct and pass in — `SessionPool`, `ConcurrencySystem`, `BrowserPool`, `RemoteBrowserPool`, `RenderingTypePredictor` — is yours to shut down. All of them implement `Symbol.asyncDispose`, so `await using` does it for you:++```typescript+await using concurrencySystem = new ConcurrencySystem({ maxConcurrency: 20 });+await concurrencySystem.start();++await Promise.all([a.run(), b.run()]);+```++The hook calls the same `stop()` / `teardown()` / `destroy()` method as before, and those stay — `await using` needs Node.js 24, and v4 supports Node.js 22.+### TypeScript 5.8+ requiredSupport for older TypeScript versions was dropped. Crawlee ships compiled JavaScript, so this only affects type-checking against its type declaration files — plain JavaScript projects are unaffected. In particular, a CJS TypeScript project needs TypeScript 5.8+ to type-check a `require()` of an ESM package like Crawlee; older versions might still work if your project is also ESM.@@ -1759,6 +1772,8 @@ try {}```+On Node.js 24, `await using` replaces the `try`/`finally` — see [collaborators you own are disposable](#collaborators-you-own-are-disposable).+#### `AutoscaledPool` is no longer public API`AutoscaledPool` is `@internal` in v4, along with `AutoscaledPoolOptions`. It is still exported from `@crawlee/core` (and re-exported by `crawlee`), so nothing breaks at import time — but with all the configuration moved to the `ConcurrencySystem`, what remains is a bare parallel task runner. It can change without a major bump, so avoid depending on it; if you only wanted bounded parallelism, a `p-limit`-style helper is a better fit than an internal Crawlee class.diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.tsindex 8c78ef8e7..ef39cbe12 100644--- a/packages/browser-pool/src/browser-pool.ts+++ b/packages/browser-pool/src/browser-pool.ts@@ -768,6 +768,10 @@ export class BrowserPool<await Promise.all(promises);}+ async [Symbol.asyncDispose](): Promise<void> {+ await this.destroy();+ }+/*** Closes all managed browsers and tears down the pool.*/diff --git a/packages/browser-pool/src/remote-browser-pool.ts b/packages/browser-pool/src/remote-browser-pool.tsindex ca743c8cc..e365766c4 100644--- a/packages/browser-pool/src/remote-browser-pool.ts+++ b/packages/browser-pool/src/remote-browser-pool.ts@@ -289,6 +289,10 @@ export class RemoteBrowserPool<Page = unknown> implements IBrowserPool<Page> {return this.#pool.injectPageState(page, state);}+ async [Symbol.asyncDispose](): Promise<void> {+ await this.destroy();+ }+/** Closes all browsers, releases any still-open remote sessions, and tears down the wrapped pool. */async destroy(): Promise<void> {await this.browserPool.destroy();diff --git a/packages/core/src/autoscaling/concurrency_system.ts b/packages/core/src/autoscaling/concurrency_system.tsindex 2fdc39776..898b7b00a 100644--- a/packages/core/src/autoscaling/concurrency_system.ts+++ b/packages/core/src/autoscaling/concurrency_system.ts@@ -409,6 +409,10 @@ export class ConcurrencySystem implements IConcurrencySystem {this.#running = true;}+ async [Symbol.asyncDispose](): Promise<void> {+ await this.stop();+ }+/*** Stops the snapshotter and intervals. Idempotent and safe to call even if the system was never started.*/diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.tsindex 3b5ec298a..7f7a71220 100644--- a/packages/core/src/session_pool/session_pool.ts+++ b/packages/core/src/session_pool/session_pool.ts@@ -381,6 +381,10 @@ export class SessionPool implements ISessionPool {);}+ async [Symbol.asyncDispose](): Promise<void> {+ await this.teardown({ persistState: true });+ }+/*** Removes listener from `persistState` event.* This function should be called after you are done with using the `SessionPool` instance.diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.tsindex f0d30a425..ca6a5319c 100644--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts@@ -820,6 +820,9 @@ export class AdaptivePlaywrightCrawler<override async teardown() {await super.teardown();+ // Mirrors the owned-only `initialize()` in `init()` - without this, the predictor we built keeps its+ // PERSIST_STATE listener registered after the crawl and never gets a final write.+ await this.#renderingTypePredictor.ifOwned((predictor) => predictor.teardown());for (const hook of this.#teardownHooks) {await hook();}diff --git a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.tsindex aece5ffa1..dfa2024f7 100644--- a/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts+++ b/packages/playwright-crawler/src/internals/utils/rendering-type-prediction.ts@@ -139,6 +139,17 @@ export class RenderingTypePredictor implements IRenderingTypePredictor {await this.state.initialize();}+ /**+ * Stop persisting the model, writing it out one last time. `initialize()` reopens the persistence window.+ */+ async teardown(): Promise<void> {+ await this.state.teardown();+ }++ async [Symbol.asyncDispose](): Promise<void> {+ await this.teardown();+ }+/*** Predict the rendering type for a given URL and request label.*/diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.tsindex 8d4ec7239..fd76ca059 100644--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts@@ -900,15 +900,33 @@ describe('AdaptivePlaywrightCrawler', () => {await expect(store.getValue<string>('rendering-type-predictor-state')).resolves.not.toBeNull();});- test('does not initialize an injected predictor', async () => {+ test('tears down the predictor it built itself', async () => {+ const crawler = new AdaptivePlaywrightCrawler({+ requestHandler,+ renderingTypeDetectionRatio: 1,+ maxConcurrency: 1,+ maxRequestRetries: 0,+ maxRequestsPerCrawl: 1,+ requestList: await RequestList.open({ sources: [`http://${HOSTNAME}:${port}/static`] }),+ });++ await crawler.run();++ // The predictor keeps a PERSIST_STATE listener from the moment it is initialized, so a leftover+ // listener after the run means the owned predictor was never torn down.+ expect(serviceLocator.getEventManager().listenerCount(EventType.PERSIST_STATE)).toBe(0);+ });++ test('does not initialize or tear down an injected predictor', async () => {const renderingTypePredictor = {...makeRiggedRenderingTypePredictor({detectionProbabilityRecommendation: 0,renderingType: 'static',}),- // Not part of the predictor contract the crawler depends on - a borrowed instance is set up by- // whoever created it, so the crawler must keep its hands off.+ // Not part of the predictor contract the crawler depends on - a borrowed instance is set up (and+ // disposed of) by whoever created it, so the crawler must keep its hands off.initialize: vi.fn(async () => {}),+ teardown: vi.fn(async () => {}),};const crawler = await makeOneshotCrawler({ requestHandler, renderingTypePredictor }, [@@ -919,6 +937,7 @@ describe('AdaptivePlaywrightCrawler', () => {expect(renderingTypePredictor.predict).toHaveBeenCalledOnce();expect(renderingTypePredictor.initialize).not.toHaveBeenCalled();+ expect(renderingTypePredictor.teardown).not.toHaveBeenCalled();});});diff --git a/test/core/crawlers/rendering_type_predictor.test.ts b/test/core/crawlers/rendering_type_predictor.test.tsindex 5c55d28f8..dc7ea4335 100644--- a/test/core/crawlers/rendering_type_predictor.test.ts+++ b/test/core/crawlers/rendering_type_predictor.test.ts@@ -1,4 +1,4 @@-import { KeyValueStore, MemoryStorageBackend, Request, serviceLocator } from '@crawlee/core';+import { EventType, KeyValueStore, MemoryStorageBackend, Request, serviceLocator } from '@crawlee/core';import { RenderingTypePredictor } from '@crawlee/playwright';import { beforeEach, describe, expect, it } from 'vitest';@@ -59,5 +59,25 @@ describe('RenderingTypePredictor', () => {expect(prediction.renderingType).toBe('clientOnly');expect(prediction.detectionProbabilityRecommendation).toBe(1);});++ it('should persist state and stop listening on teardown', async () => {+ const persistStateKey = 'rendering-type-predictor-teardown';+ const events = serviceLocator.getEventManager();++ const predictor = new RenderingTypePredictor({+ detectionRatio: 0.1,+ persistenceOptions: { persistStateKey },+ });+ await predictor.initialize();+ predictor.storeResult(new Request({ url: 'https://example.com/static-page' }), 'static');++ const listenersBefore = events.listenerCount(EventType.PERSIST_STATE);++ await predictor.teardown();++ expect(events.listenerCount(EventType.PERSIST_STATE)).toBe(listenersBefore - 1);+ const store = await KeyValueStore.open();+ expect(await store.getValue(persistStateKey)).toHaveProperty('detectionResults');+ });});});diff --git a/tsconfig.build.json b/tsconfig.build.jsonindex 39fd085ae..26cf6a86c 100644--- a/tsconfig.build.json+++ b/tsconfig.build.json@@ -4,7 +4,7 @@"module": "NodeNext","moduleResolution": "NodeNext","target": "ESNext",- "lib": ["DOM", "ES2023", "ES2024", "DOM.AsyncIterable"],+ "lib": ["DOM", "ES2023", "ES2024", "DOM.AsyncIterable", "ESNext.Disposable"],"types": ["node"],"allowJs": true,"skipLibCheck": true,