Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
refactor!: Introduce IBrowserPool interface and utilize it in BrowserCrawler (#3669)
Jan Buchar97701bd499ce
diff --git a/REBASE-DEFERRED.md b/REBASE-DEFERRED.mdnew file mode 100644index 000000000..0e0fbf4a4--- /dev/null+++ b/REBASE-DEFERRED.md@@ -0,0 +1,295 @@+# Rebase deferrals — reconcile against `v4-reverse` before finishing++This file tracks resolutions made during the `v4` → `master` rebase that were+**not** brought fully to the desired end state (`v4-reverse`). Untracked on+purpose — delete once reconciled. Do a final `git diff <final-HEAD> v4-reverse`+per file to confirm nothing below was lost.++## `packages/http-crawler/src/internals/file-download.ts`++**Commit:** `cf72dda66` — `refactor!: Introduce the ContextPipeline abstraction (#3119)`++**What I did:** took #3119's coherent committed version of the whole file+(`git checkout --theirs`). The 3 conflict hunks were interdependent and 7 later+rebase commits reshape this file, so a hunk-by-hunk merge risked dangling+references.++**Deferred (present in `v4-reverse`, NOT re-added by any rebase commit — must be+folded back at the end):**+- Master's typed schema-router overload: `RouteSchemas` / `RoutesFromSchemas`+ imports from `../index.js`, and the `downloadFile`/router-factory overload that+ returns `RouterHandler<Context, RoutesFromSchemas<Schemas>>`. Type-level only,+ no runtime effect, compiles fine without it.+- Confirm `abortDownload` handling matches `v4-reverse` (v4-reverse keeps it; it+ may arrive via a later commit — verify it isn't dropped).++**Reconcile with:**+`git diff <final-HEAD>:packages/http-crawler/src/internals/file-download.ts \+ v4-reverse:packages/http-crawler/src/internals/file-download.ts`++## Master-only features that WERE folded in (not lost — listed for verification)++- `playwright-crawler.ts` `enhanceContext`: `listDownloads` (downloads array ++ `page.on('download')` + `listDownloads: async () => downloads`). Master-only,+ no rebase commit re-adds it; folded in to match `v4-reverse`.+- `browser-crawler.ts` `buildContextPipeline` cleanup: puppeteer-25+ `addTimeoutToPromise` wrapper around `page.close()`.+- `browser-crawler.ts`: kept master's `userRequestHandler` getter override.+- `adaptive-playwright-crawler.ts`: kept master's `shouldPropagateError` check,+ combined with #3119's `RequestHandlerError.cause` unwrap.++## General++Recurring mechanical conflicts (ESM `.js` imports, `test/shared/*` paths,+`3.17.0` version bumps, `RequestValidationError`/`zod` schema-feature imports)+are auto-applied by git rerere. Verify each rerere-resolved commit is+marker-free; spot-checked so far and clean.++## Possible duplicate import to verify — `adaptive-playwright-crawler.ts`++At/after the ContextPipeline commit, the top of+`packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts` may+import `BrowserHook` / `LoadedRequest` / `Request` from `@crawlee/browser` on+BOTH an early `import type {...}` block and a later+`import type { BasicCrawlerOptions, BrowserHook, LoadedRequest, Request } from '@crawlee/browser'`.+Duplicate type imports from the same module = TS error. Check the final tree and+dedupe if present (a later rebase commit may already fix it).++## Latent bug — `packages/core/src/storages/request_list.ts` dangling `this.events`++Commit `ba3a3568a` (`refactor!: Extract service management from Configuration+into ServiceLocator class #3325`) removes the `private events: EventManager`+field and its `this.events = config.getEventManager()` assignment, replacing+the one conflicting usage (`initialize()`'s `.on(EventType.PERSIST_STATE, ...)`+call) with `serviceLocator.getEventManager().on(...)`.++However, a master-only teardown method (not part of the conflict, already+auto-merged in) still calls `this.events.off(EventType.PERSIST_STATE, ...)`+around line 542 — `events` no longer exists on the class, so this is a+TypeScript compile error (`Property 'events' does not exist`).++**Confirmed this is not just my resolution**: `v4-reverse` has the identical+dangling `this.events.off(...)` call at its equivalent line. No commit further+along the rebase touches it either. This needs a manual fix — swap it to+`serviceLocator.getEventManager().off(EventType.PERSIST_STATE, this.persistState)`+— whenever the tree is type-checked.++## ServiceLocator commit (`ba3a3568a` #3325) — architecture collision in Snapshotter/SystemStatus++Master had already refactored `Snapshotter`/`SystemStatus` into a composable+`LoadSignal` architecture (memory/cpu/event-loop/client signal objects with+`.start()/.stop()/.handle()`) in an earlier-applied commit. `ba3a3568a` is v4's+older, pre-LoadSignal, monolithic interval-based Snapshotter rewritten only for+ServiceLocator. These two are fundamentally different implementations of the+same class — verified against `v4-reverse` that the LoadSignal architecture+survives, so for every conflicting method (imports, class fields, `start()`,+`stop()`, `_snapshotClient`/`handle()`) I kept HEAD's LoadSignal delegation and+discarded theirs' interval/serviceLocator-direct logic entirely.++**Found and fixed a real latent bug in the process**: `Snapshotter`'s+constructor had its `client`/`config` destructuring and `this.client =+/this.config =` assignments silently dropped by an earlier (non-conflicting)+auto-merge upstream in this rebase, while `this.client`/`this.config` were+still referenced further down (passed into the `MemoryLoadSignal`/+`createCpuLoadSignal`/`createClientLoadSignal` factories, which — at this point+in history — still require them as explicit params; they don't yet pull from+`serviceLocator` internally). Restored the wiring using commit `308da2263`+(an early point in this rebase where it was still intact) as reference,+adapted to call `serviceLocator.getConfiguration()` /+`serviceLocator.getStorageClient()` as defaults (matching this commit's+intent) instead of the old `Configuration.getGlobalConfig()` /+`config.getStorageClient()`. Also switched the `Configuration` import to+`import type` since it's now type-only in this file.++`SystemStatus` needed the same reconciliation: kept master's `loadSignals`+option (composes custom signals with the snapshotter's built-in ones) but+dropped the `config` option/field entirely — `new Snapshotter()` with no args+now works since the above fix wires its own `serviceLocator` defaults.++`storage_manager.ts` had a similar merge artifact: an unconflicted `try/finally`+wrapped body (from a separate already-applied commit) duplicated by this+commit's non-try/finally version of the same logic, producing a dead+duplicate tail after the conflict markers. Deleted the duplicate and applied+this commit's actual semantic change (`this.config.getStorageClient()` →+`serviceLocator.getStorageClient()`) to the surviving copy.++**Please re-verify** the `Snapshotter`/`SystemStatus`/`storage_manager.ts`+resolutions once dependencies are installed and `tsc`/tests can run — these+were reasoned through by reading surrounding code and cross-checking+`v4-reverse`, not verified by a compiler.++## Fixed a real dead-function bug — `local_event_manager.ts` `getMemoryInfoV2`++While resolving commit `7c3ba07ea` ("refactor: resolve last direct @apify/log+calls"), found that `LocalEventManager`'s private `getMemoryInfo()` helper+branched on `this.config.get('systemInfoV2')` to call `getMemoryInfoV2(...)` —+but **`getMemoryInfoV2` does not exist anywhere in `@crawlee/utils`** (only+`getMemoryInfo` and `getCurrentCpuTicksV2` exist; the CPU side got a real V2+implementation, the memory side never did). That branch would throw+`ReferenceError` at runtime for anyone with `systemInfoV2` config enabled.++Also found the surviving (non-V2) branch's `getMemoryInfo()` call had no+import at all in scope — a dangling reference from an earlier upstream+auto-merge, same class of bug as the `request_list.ts` one documented above.++**Fix applied**: collapsed the private `getMemoryInfo()` method to just the+working path — dynamically imports `getMemoryInfo` from `@crawlee/utils` and+passes `containerized`/`logger`, matching this commit's intent and the+pattern already used by the sibling `createCpuInfo()` method (which does have+a working V2 path via `getCurrentCpuTicksV2`). Dropped the dead+`systemInfoV2`/`getMemoryInfoV2` branch entirely rather than inventing a+`getMemoryInfoV2` implementation, which would be scope creep beyond a merge+conflict fix. `LocalEventManager.prototype.getMemoryInfo` is still spied on by+`test/core/autoscaling/snapshotter.test.ts`, so the method itself is kept —+only its broken internals were fixed.++**Please verify**: if a memory-specific V2 code path was intended (mirroring+`getCurrentCpuTicksV2`), it needs to be implemented for real — this fix does+not add one, it just removes a dead reference to a non-existent one.++## yarn → pnpm migration (`930b2ef4f`) — needs verification with a real `pnpm install`++Per your explicit choice, I took theirs (pnpm) fully for this commit: deleted+`yarn.lock`, `docs/yarn.lock`, `website/yarn.lock`, `.yarnrc.yml`; adopted+`[email protected]` as `packageManager`/volta; rewrote CI workflow yarn/corepack+steps to `apify/workflows/pnpm-install@main`; kept master's newer+devDependency versions (`@apify/tsconfig`, `@commitlint`, `@playwright/browser-*`,+`typescript`, `playwright`, docusaurus 3.10.2) alongside the new oxlint/oxfmt ++pnpm tooling.++**One thing I could not verify mechanically**: master's root `package.json`+had a yarn-only `"resolutions"` block (`tmp`, `@puppeteer/browsers`,+`[email protected]`, `form-data`, `tar`, `lerna/js-yaml`) that this+commit drops in favor of pnpm's `overrides` in `pnpm-workspace.yaml` (which+already carried v4's own `[email protected]`, `@browserbasehq/stagehand`,+`minimatch` overrides). I merged master's entries into that `overrides:`+block, bumping `playwright-core` to `1.61.1` to match the `playwright`+devDependency version we kept (a version mismatch between the two would be+worse than dropping the override). For the yarn-specific nested-scope+override `lerna/js-yaml`, I translated it to pnpm's `"lerna>js-yaml"` syntax+— **this translation is unverified**; pnpm's override key syntax has specific+rules I can't test without actually running `pnpm install` (which requires+installing the new package manager). Please run `pnpm install` and confirm+the lockfile resolves cleanly, especially the `js-yaml` override under+`lerna`.++## `storage_manager.ts` `openStorage` — kept try/finally around the new StorageClient logic++Commit `ffff3347e` (`refactor!: Overhaul the StorageClient interface #3570`)+rewrites `StorageManager.openStorage` to resolve identifiers via+`_resolveIdentifier`/`_createSubClient` (both already present unconflicted+elsewhere in the file). Its own version of the method does NOT wrap the body+in `try/finally` — `this.storageOpenQueue.shift()` runs as a plain trailing+statement. The pre-this-commit version (which I'd already resolved in an+earlier conflict) wrapped the equivalent logic in `try { ... } finally {+this.storageOpenQueue.shift(); }` specifically so the queue lock is always+released even if resolution/creation throws.++I kept the `try/finally` safety net around theirs' new logic rather than+taking the method verbatim, since dropping it reintroduces a real deadlock+risk (a failed `_resolveIdentifier`/`_createSubClient`/`getMetadata` call+would permanently wedge `storageOpenQueue` for all subsequent `openStorage`+calls on that manager). This wasn't verifiable against `v4-reverse` (that+file is renamed/rewritten further into `storage_instance_manager.ts` with an+unrelated alias-based architecture by then). Worth a second look once you can+run the test suite.++## Fixed more dead API calls — `Configuration` redesign (`d1f4c98e5` #3484)++This commit ("feat: redesign `Configuration` class for v4") replaces+`Configuration`'s `.get(key, default)`/`.set()` accessor methods with plain+property access (e.g. `config.availableMemoryRatio` instead of+`config.get('availableMemoryRatio')`), and removes `Configuration.getEventManager()`/+`.getStorageClient()` entirely (those now only live on `serviceLocator`).++Same class of bug as the `request_list.ts`/`local_event_manager.ts` issues+documented above: two files outside this commit's own diff — never flagged+by any merge conflict — still called the now-deleted methods:++- `packages/core/src/autoscaling/memory_load_signal.ts`: `this.config.get(...)`+ (×3) and `this.config.getEventManager()`. Fixed to plain property access+ (`this.config.memoryMbytes`, `.availableMemoryRatio`, `.containerized`) and+ `serviceLocator.getEventManager()`. Also dropped the `systemInfoV2`/+ `getMemoryInfoV2` branch in `_getTotalMemoryBytes()` — confirmed+ `getMemoryInfoV2` doesn't exist anywhere in `@crawlee/utils` (same dead+ function found earlier in `local_event_manager.ts`) and `systemInfoV2`+ isn't a field on the new `Configuration` schema either. Restored the+ `isContainerized()` auto-detect fallback (`this.config.containerized ??+ (await isContainerized())`) to match the behavior the original code had,+ since `getMemoryInfo()`'s own default for missing `containerized` is `false`,+ not auto-detection.+- `packages/core/src/autoscaling/cpu_load_signal.ts`: `options.config.getEventManager()`+ → `serviceLocator.getEventManager()`. Left the now-unused-but-still-required+ `config: Configuration` field on `CpuLoadSignalOptions` alone rather than+ reworking the signal-creation API — `Snapshotter` still passes `config` when+ constructing this signal, so removing the field would be a wider API change+ outside the scope of this fix.++Repo-wide grep after these fixes shows no remaining `.config.get(...)`,+`.config.getEventManager()`, or `.config.getStorageClient()` calls anywhere+under `packages/*/src`. Please re-verify once the tree can be type-checked —+these were reasoned through by reading the new `Configuration` class and+cross-referencing the already-fixed sibling file, not compiler-verified.++## Update: `lerna>js-yaml` override syntax confirmed correct++A later commit (`36022e7c7`, "pin lerna's minimatch to v3") independently adds+`"lerna>minimatch": "^3.1.4"` to `pnpm-workspace.yaml`'s `overrides`, with a+comment explaining lerna's bundled code needs the CJS-style v3 export. This+confirms the `"packageA>packageB"` scoped-override syntax I used for+`lerna>js-yaml` (flagged as unverified above) is exactly right — no longer a+concern.++## `e00aa9419` — DatasetClient/KeyValueStoreClient aligned with Python (rename support removed)++This commit removes `update()` (rename-a-storage support) entirely from+`DatasetClient`/`KeyValueStoreClient`/`RequestQueueClient` in memory-storage,+renames `delete()` → `drop()`, and adds `purge()`. Confirmed via+`packages/types/src/storages.ts`: the interface no longer declares `update()`+at all, only `drop()`/`purge()` — so removing the implementations was correct,+not a data-loss risk.++For each of the three resource-client files I dropped the `update()` method+body along with the `resolveWithinDirectory`/`move`-based rename logic it+contained, kept the constructor's own `resolveWithinDirectory(...)` call+(master's path-traversal hardening — confirmed via `git show+e00aa9419:...` that v4's own version of these files uses a bare `resolve()`+in the constructor too, so this hardening predates/is independent of this+commit and must survive it), and adopted `drop()`. Also removed now-dead+imports (`createKeyList`/`createKeyStringList`/`createLazyIterablePromise` in+key-value-store.ts — `keys()`/`values()`/`entries()` no longer exist on the+class; `Readable`, `move`, `StorageTypes` similarly unused post-removal).++**`packages/memory-storage/test/async-iteration.test.ts`**: HEAD's version+tested `kvStore.keys()`/`.values()`/`.entries()` and imported+`createLazyIterablePromise` — none of which exist anymore (confirmed via+grep on the resolved `key-value-store.ts`). Rather than attempt a partial+merge of an obsolete test suite, took theirs' version of the whole file+wholesale (`git checkout --theirs`) — it tests the new `getData()`/+`listKeys()` API directly and is self-contained.++**`packages/memory-storage/test/request-queue/handledRequestCount-should-update.test.ts`**:+HEAD added two new tests. One ("deleting a request should decrement...")+calls `requestQueue.deleteRequest(id)`, which doesn't exist on the current+`RequestQueueClient` (no per-request delete method at all in this interface,+confirmed via grep of all `async` methods) — dropped, it tests a capability+that isn't there. The other ("updating an already handled request should not+increment...") calls `requestQueue.get()` (renamed to `getMetadata()`) but is+otherwise valid and valuable — it's the regression test for the exact+`isRequestHandledStateChanging` double-count fix preserved during an earlier+conflict in this same rebase (commit `20b320add`'s `request-queue.ts`+conflict). Kept it, renamed `get()` → `getMetadata()`.++**Commit `ebb1b2632` (Introduce IBrowserPool interface) — moved the puppeteer-25+`page.close()` timeout guard**: master (`1430062c2`) had wrapped+`page.close()` in `browser-crawler.ts` with a 5s `addTimeoutToPromise` guard+(puppeteer 25 can hang `page.close()` indefinitely after an aborted+navigation). This v4 commit replaces that direct `page.close()` call with+`this.browserPool.closePage(context.page, { error })`, moving page-closing+into the pool abstraction. To keep both fixes, moved the timeout guard itself+into `browser-pool.ts`'s `closePage()` method (added `PAGE_CLOSE_TIMEOUT_MILLIS`+there, imported `addTimeoutToPromise`) rather than doubly closing the page+from both `browser-crawler.ts` and the pool. Removed the now-dead+`PAGE_CLOSE_TIMEOUT_MILLIS` constant from `browser-crawler.ts` (no other use).+Worth double-checking in review that no other caller of `closePage()` relies+on it settling faster than 5s.diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex b9f1e61b3..67a965ba3 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -214,6 +214,70 @@ const crawler = new BasicCrawler({});```+## Custom `BrowserPool` implementations via the `IBrowserPool` interface++Browser crawlers now accept any object implementing the new `IBrowserPool` interface as their `browserPool` option, not just instances of the built-in `BrowserPool`. The interface follows the classic acquire/release pattern, plus a pair of helpers for moving state between the crawling session and the page:++- **`newPage(options?)`** — opens a new page. An optional `session` can be passed as a best-effort hint — the pool may use it for proxy configuration, fingerprinting, etc., but nothing is guaranteed.+- **`closePage(page, options?)`** — signals the pool that the caller is done with the page. If the optional `error` is a `SessionError`, the pool should purge all state associated with the session (e.g. retire the underlying browser).+- **`extractPageState(page)`** — reads the relevant state (currently cookies) out of a page so the crawler can persist it back into the session.+- **`injectPageState(page, state)`** — the counterpart to `extractPageState`; seeds a page with state (currently cookies) before navigation. Isolation between pages is best-effort and depends on the pool implementation.++Lifecycle (`destroy`) is the responsibility of whoever owns the pool: a custom pool you construct yourself is never owned by the crawler, so the crawler never tears it down. This makes it straightforward to plug in a remote browser farm, a session-aware pool, or another custom browser-management strategy without subclassing `BrowserPool`.++```typescript+import { PuppeteerCrawler } from '@crawlee/puppeteer';+import { BrowserPool, PuppeteerPlugin, type IBrowserPool } from '@crawlee/browser-pool';+import puppeteer from 'puppeteer';++const sharedPool = new BrowserPool({ browserPlugins: [new PuppeteerPlugin(puppeteer)] });++const crawler = new PuppeteerCrawler({+ browserPool: sharedPool,+ requestHandler: async ({ page }) => {+ // …+ },+});++// You own `sharedPool` — destroy it yourself when you're done.+await crawler.run();+await sharedPool.destroy();+```++## `BrowserCrawlingContext.browserController` has been removed++The `browserController` property is no longer part of the crawling context (`BrowserCrawlingContext`). Browser controller management is now fully internal to the pool — the crawler interacts with the pool only through the `IBrowserPool` interface (`newPage`, `closePage`, `extractPageState`, and `injectPageState`).++If you previously used `browserController` in your request handlers, here is how to migrate the most common patterns:++**Cookies** — Cookie injection and persistence are now handled automatically by the crawler and the pool. You no longer need to call `browserController.getCookies()` or `browserController.setCookies()` manually.++**Proxy info** — Access proxy information via `session.proxyInfo` instead of `browserController.launchContext.proxyUrl`. TLS-error handling moved along with it: the pool reads `session.proxyInfo.ignoreTlsErrors`, so there is no standalone `ignoreTlsErrors` page option anymore. If you need to disable TLS verification for some other reason, set `ignoreHTTPSErrors` (Playwright) / `acceptInsecureCerts` (Puppeteer) through the browser's `launchOptions`.++**Direct browser access** — If you need the raw browser or controller instance (e.g. for Puppeteer/Playwright-specific APIs), construct a `BrowserPool` yourself, pass it to the crawler, and reference it directly in your handler — no cast needed:++```typescript+import { BrowserPool, PuppeteerPlugin } from '@crawlee/browser-pool';+import { PuppeteerCrawler } from '@crawlee/puppeteer';+import puppeteer from 'puppeteer';++const pool = new BrowserPool({ browserPlugins: [new PuppeteerPlugin(puppeteer)] });++const crawler = new PuppeteerCrawler({+ browserPool: pool,+ requestHandler: async ({ page }) => {+ const controller = pool.getBrowserControllerByPage(page);+ // controller.browser, controller.launchContext, etc.+ },+});++await crawler.run();+// You own the pool — tear it down yourself.+await pool.destroy();+```++Note that this couples your code to the built-in `BrowserPool` — custom `IBrowserPool` implementations may not expose controllers at all.+## `tieredProxyUrls` is removed from `ProxyConfiguration`The `tieredProxyUrls` option has been removed, together with the `proxyTier` field on `ProxyInfo` and the `proxyTier` plumbing in `BrowserPool`. In v4 the `Session` is the main rotation unit - a session already carries its own proxy, cookies and error score, so the pool rotates the whole fingerprint when a session gets retired on a block.diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.tsindex 7997c3deb..8299b990c 100644--- a/packages/browser-crawler/src/internals/browser-crawler.ts+++ b/packages/browser-crawler/src/internals/browser-crawler.ts@@ -38,7 +38,7 @@ import type {LaunchContext,} from '@crawlee/browser-pool';import { BrowserPool } from '@crawlee/browser-pool';-import type { BatchAddRequestsResult, Cookie as CookieObject, ISession } from '@crawlee/types';+import type { BatchAddRequestsResult, Cookie as CookieObject, IBrowserPool, ISession } from '@crawlee/types';import type { RobotsTxtFile } from '@crawlee/utils';import { CLOUDFLARE_RETRY_CSS_SELECTORS, RETRY_CSS_SELECTORS, sleep } from '@crawlee/utils';import ow from 'ow';@@ -48,8 +48,6 @@ import { tryCancel } from '@apify/timeout';import type { BrowserLaunchContext } from './browser-launcher.js';-const PAGE_CLOSE_TIMEOUT_MILLIS = 5000;-interface BaseResponse {status(): number;}@@ -59,15 +57,9 @@ type ContextDifference<T, U> = Omit<U, keyof T> & Partial<U>;export interface BrowserCrawlingContext<Page extends CommonPage = CommonPage,Response extends BaseResponse = BaseResponse,- ProvidedController = BrowserController,UserData extends Dictionary = Dictionary,GoToOptions extends Dictionary = Dictionary,> extends CrawlingContext<UserData> {- /**- * An instance of the {@apilink BrowserController} that manages the browser instance and provides access to its API.- */- browserController: ProvidedController;-/*** The browser page object where the web page is loaded and rendered.*/@@ -106,11 +98,9 @@ const readContextField = <T>(ctx: object, key: symbol): T => (ctx as Record<symbexport interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage,Response extends BaseResponse = BaseResponse,- ProvidedController extends BrowserController = BrowserController,- Context extends BrowserCrawlingContext<Page, Response, ProvidedController, Dictionary> = BrowserCrawlingContext<+ Context extends BrowserCrawlingContext<Page, Response, Dictionary> = BrowserCrawlingContext<Page,Response,- ProvidedController,Dictionary>,ContextExtension = Dictionary<never>,@@ -126,6 +116,13 @@ export interface BrowserCrawlerOptions<> {launchContext?: BrowserLaunchContext<any, any>;+ /**+ * An existing browser pool instance to use. When provided, the crawler will use this pool directly instead of+ * constructing a new one from `browserPoolOptions`, enabling browser sharing across multiple crawlers. The crawler+ * will not tear down a shared pool — the caller is responsible for its lifecycle.+ */+ browserPool?: IBrowserPool<Page>;+/*** Function that is called to process each request.*@@ -136,7 +133,6 @@ export interface BrowserCrawlerOptions<* - {@apilink BrowserCrawlingContext.page|`page`} is an instance of the* Puppeteer [Page](https://pptr.dev/api/puppeteer.page) or* Playwright [Page](https://playwright.dev/docs/api/class-page);- * - {@apilink BrowserCrawlingContext.browserController|`browserController`} is an instance of the {@apilink BrowserController};* - {@apilink BrowserCrawlingContext.response|`response`} is an instance of the* Puppeteer [Response](https://pptr.dev/api/puppeteer.httpresponse) or* Playwright [Response](https://playwright.dev/docs/api/class-response),@@ -308,13 +304,11 @@ export interface BrowserCrawlerOptions<export abstract class BrowserCrawler<Page extends CommonPage = CommonPage,Response extends BaseResponse = BaseResponse,- ProvidedController extends BrowserController = BrowserController,InternalBrowserPoolOptions extends BrowserPoolOptions = BrowserPoolOptions,LaunchOptions extends Dictionary | undefined = Dictionary,- Context extends BrowserCrawlingContext<Page, Response, ProvidedController, Dictionary> = BrowserCrawlingContext<+ Context extends BrowserCrawlingContext<Page, Response, Dictionary> = BrowserCrawlingContext<Page,Response,- ProvidedController,Dictionary>,ContextExtension = Dictionary<never>,@@ -322,9 +316,18 @@ export abstract class BrowserCrawler<GoToOptions extends Dictionary = Dictionary,> extends BasicCrawler<Context, ContextExtension, ExtendedContext> {/**- * A reference to the underlying {@apilink BrowserPool} class that manages the crawler's browsers.+ * A reference to the underlying browser pool that manages the crawler's browsers. Typed as+ * {@apilink IBrowserPool} so custom implementations can be plugged in via the `browserPool` constructor option.+ */+ browserPool: IBrowserPool<Page>;++ /**+ * Set when the crawler constructed its own {@apilink BrowserPool} (no `browserPool` option was provided).+ * Holds the same instance as `browserPool`, but typed as the concrete class so the crawler can call+ * lifecycle methods (`destroy`) that aren't part of {@apilink IBrowserPool}. A user-supplied pool is+ * never owned and never torn down by the crawler.*/- browserPool: BrowserPool<InternalBrowserPoolOptions>;+ private ownedBrowserPool?: BrowserPool<InternalBrowserPoolOptions>;launchContext: BrowserLaunchContext<LaunchOptions, unknown>;@@ -345,7 +348,8 @@ export abstract class BrowserCrawler<launchContext: ow.optional.object,headless: ow.optional.any(ow.boolean, ow.string),- browserPoolOptions: ow.object,+ browserPool: ow.optional.object.validate(validators.browserPool),+ browserPoolOptions: ow.optional.object,saveResponseCookies: ow.optional.boolean,proxyConfiguration: ow.optional.object.validate(validators.proxyConfiguration),};@@ -354,14 +358,7 @@ export abstract class BrowserCrawler<* All `BrowserCrawler` parameters are passed via an options object.*/protected constructor(- options: BrowserCrawlerOptions<- Page,- Response,- ProvidedController,- Context,- ContextExtension,- ExtendedContext- > & {+ options: BrowserCrawlerOptions<Page, Response, Context, ContextExtension, ExtendedContext> & {contextPipelineBuilder: () => ContextPipeline<CrawlingContext, Context>;},) {@@ -370,6 +367,7 @@ export abstract class BrowserCrawler<navigationTimeoutSecs = 60,saveResponseCookies = true,launchContext = {},+ browserPool,browserPoolOptions,preNavigationHooks = [],postNavigationHooks = [],@@ -424,46 +422,46 @@ export abstract class BrowserCrawler<this.saveResponseCookies = saveResponseCookies;+ if (browserPool) {+ this.browserPool = browserPool;+ return;+ }++ const resolvedBrowserPoolOptions = browserPoolOptions ?? ({} as Partial<BrowserPoolOptions>);+if (launchContext?.userAgent) {- if (browserPoolOptions.useFingerprints)+ if (resolvedBrowserPoolOptions.useFingerprints)this.log.info('Custom user agent provided, disabling automatic browser fingerprint injection!');- browserPoolOptions.useFingerprints = false;+ resolvedBrowserPoolOptions.useFingerprints = false;}- this.browserPool = new BrowserPool<InternalBrowserPoolOptions>({- ...(browserPoolOptions as any),+ this.ownedBrowserPool = new BrowserPool<InternalBrowserPoolOptions>({+ ...(resolvedBrowserPoolOptions as any),});+ this.browserPool = this.ownedBrowserPool as IBrowserPool<Page>;}protected override buildContextPipeline(): ContextPipeline<CrawlingContext,- BrowserCrawlingContext<Page, Response, ProvidedController, Dictionary>+ BrowserCrawlingContext<Page, Response, Dictionary>> {return ContextPipeline.create<CrawlingContext>().compose({action: this.preparePage.bind(this),cleanup: async (context: {page: Page;session: ISession;- browserController: ProvidedController;registerDeferredCleanup: BasicCrawlingContext['registerDeferredCleanup'];}) => {context.registerDeferredCleanup(async () => {- // In non-incognito mode the browser controller carries the session's cookies- // and storage across pages. If the session is no longer usable, retire the- // controller so its state can't leak into whichever session lands on it next.- if (!context.session.isUsable()) {- this.browserPool.retireBrowserController(- context.browserController as Parameters<- BrowserPool<InternalBrowserPoolOptions>['retireBrowserController']- >[0],+ const error = !context.session.isUsable()+ ? new SessionError('Session is no longer usable')+ : undefined;++ await this.browserPool+ .closePage(context.page, { error })+ .catch((closeError: Error) =>+ this.log.debug('Error while closing page', { error: closeError }),);- }- // Puppeteer 25+ can hang `page.close()` indefinitely when the page's navigation was aborted, don't let it block the crawler.- await addTimeoutToPromise(- async () => context.page.close(),- PAGE_CLOSE_TIMEOUT_MILLIS,- `page.close() timed out after ${PAGE_CLOSE_TIMEOUT_MILLIS / 1000} seconds`,- ).catch((error: Error) => this.log.debug('Error while closing page', { error }));});},});@@ -478,9 +476,7 @@ export abstract class BrowserCrawler<return foundSelectors.length > 0 ? foundSelectors : null;}- protected async isRequestBlocked(- crawlingContext: BrowserCrawlingContext<Page, Response, ProvidedController>,- ): Promise<string | false> {+ protected async isRequestBlocked(crawlingContext: BrowserCrawlingContext<Page, Response>): Promise<string | false> {const { page, response } = crawlingContext;// Cloudflare specific heuristic - wait 5 seconds if we get a 403 for the JS challenge to load / resolve.@@ -513,27 +509,13 @@ export abstract class BrowserCrawler<private async preparePage(crawlingContext: CrawlingContext,- ): Promise<- ContextDifference<CrawlingContext, BrowserCrawlingContext<Page, Response, ProvidedController, Dictionary>>- > {- const newPageOptions: Dictionary = {+ ): Promise<ContextDifference<CrawlingContext, BrowserCrawlingContext<Page, Response, Dictionary>>> {+ const page = await this.browserPool.newPage({id: crawlingContext.id,- };-- if (crawlingContext.session?.proxyInfo) {- const proxyInfo = crawlingContext.session.proxyInfo;-- newPageOptions.proxyUrl = proxyInfo?.url;- newPageOptions.ignoreTlsErrors = proxyInfo?.ignoreTlsErrors;- }-- const page = (await this.browserPool.newPage(newPageOptions)) as Page;+ session: crawlingContext.session,+ });tryCancel();- const browserControllerInstance = this.browserPool.getBrowserControllerByPage(- page as any,- ) as ProvidedController;-const contextEnqueueLinks = crawlingContext.enqueueLinks;return {@@ -546,7 +528,6 @@ export abstract class BrowserCrawler<get gotoOptions(): Dictionary {throw new Error('The `gotoOptions` property is not available until `prepareNavigation` runs.');},- browserController: browserControllerInstance,enqueueLinks: async (enqueueOptions: EnqueueLinksOptions = {}) => {return (await browserCrawlerEnqueueLinks({// `contextEnqueueLinks` clamps `limit` by the remaining `maxRequestsPerCrawl` budget itself;@@ -634,7 +615,7 @@ export abstract class BrowserCrawler<// TODO: Should we save the cookies also after/only the handle page?if (this.saveResponseCookies && crawlingContext.session) {- const cookies = await crawlingContext.browserController.getCookies(crawlingContext.page);+ const { cookies } = await this.browserPool.extractPageState(crawlingContext.page);tryCancel();const url = crawlingContext.request.loadedUrl!;for (const cookie of cookies) {@@ -651,9 +632,7 @@ export abstract class BrowserCrawler<return { request: crawlingContext.request as LoadedRequest<Request> } as Partial<Context>;}- private async handleBlockedRequestByContent(- crawlingContext: BrowserCrawlingContext<Page, Response, ProvidedController>,- ) {+ private async handleBlockedRequestByContent(crawlingContext: BrowserCrawlingContext<Page, Response>) {if (this.retryOnBlocked) {const error = await this.isRequestBlocked(crawlingContext);if (error) throw new SessionError(error);@@ -668,7 +647,7 @@ export abstract class BrowserCrawler<}protected async _applyCookies(- { session, request, page, browserController }: BrowserCrawlingContext,+ { session, request, page }: BrowserCrawlingContext<Page, Response>,preHooksCookies: string,postHooksCookies: string,) {@@ -676,12 +655,11 @@ export abstract class BrowserCrawler<const parsedPreHooksCookies = preHooksCookies.split(/ *; */).map((c) => cookieStringToToughCookie(c));const parsedPostHooksCookies = postHooksCookies.split(/ *; */).map((c) => cookieStringToToughCookie(c));- await browserController.setCookies(- page,- [...sessionCookie, ...parsedPreHooksCookies, ...parsedPostHooksCookies]- .filter((c): c is CookieObject => typeof c !== 'undefined' && c !== null)- .map((c) => ({ ...c, url: c.domain ? undefined : request.url })),- );+ const cookies = [...sessionCookie, ...parsedPreHooksCookies, ...parsedPostHooksCookies]+ .filter((c): c is CookieObject => typeof c !== 'undefined' && c !== null)+ .map((c) => ({ ...c, url: c.domain ? undefined : request.url }));++ await this.browserPool.injectPageState(page, { cookies });}/**@@ -709,7 +687,7 @@ export abstract class BrowserCrawler<}protected abstract _navigationHandler(- crawlingContext: BrowserCrawlingContext<Page, Response, ProvidedController>,+ crawlingContext: BrowserCrawlingContext<Page, Response>,gotoOptions: GoToOptions,): Promise<Context['response'] | null | undefined>;@@ -749,7 +727,7 @@ export abstract class BrowserCrawler<* @ignore*/override async teardown(): Promise<void> {- await this.browserPool.destroy();+ await this.ownedBrowserPool?.destroy();await super.teardown();}}diff --git a/packages/browser-pool/src/abstract-classes/browser-controller.ts b/packages/browser-pool/src/abstract-classes/browser-controller.tsindex cb9435085..f76bfdeb2 100644--- a/packages/browser-pool/src/abstract-classes/browser-controller.ts+++ b/packages/browser-pool/src/abstract-classes/browser-controller.ts@@ -12,6 +12,76 @@ import type { BrowserPlugin, CommonBrowser, CommonLibrary } from './browser-plugconst PROCESS_KILL_TIMEOUT_MILLIS = 5000;+/**+ * The subset of the browser-pool `LaunchContext` that {@apilink IBrowserController} exposes.+ * Other fields are only available on the concrete `LaunchContext` class.+ */+export interface IBrowserLaunchContext {+ /**+ * The proxy URL the browser was launched with, if any.+ */+ proxyUrl?: string;+ /**+ * The fingerprint applied to the browser, if fingerprinting is enabled.+ * Typed as `unknown` here; cast to the concrete `LaunchContext` if you+ * need the structured shape.+ */+ fingerprint?: unknown;+ /**+ * `true` if each page in this browser uses its own context.+ */+ useIncognitoPages?: boolean;+ /**+ * The actual options the browser was launched with, after pre-launch hooks.+ */+ launchOptions?: Dictionary | undefined;+}++/**+ * The minimal public contract of a browser controller.+ *+ * Coordination with the pool (page-counting, `activate`, `assignBrowser`, lifecycle+ * promises, …) is intentionally **not** part of this contract.+ *+ * @category Browser management+ */+export interface IBrowserController<Page = unknown> {+ /**+ * A stable identifier for this controller instance. Useful for tracking+ * which browser served which request.+ */+ readonly id: string;++ /**+ * The configuration the underlying browser was launched with — proxy URL,+ * fingerprint, session, launcher-specific options, etc.+ */+ readonly launchContext: IBrowserLaunchContext;++ /**+ * The raw browser handle from the underlying automation library+ * (Puppeteer `Browser`, Playwright `Browser`/`BrowserContext`, …).+ * Escape hatch for things the controller does not expose directly.+ */+ readonly browser: unknown;++ /**+ * Reads cookies for the given page.+ */+ getCookies(page: Page): Promise<Cookie[]>;++ /**+ * Writes cookies for the given page.+ */+ setCookies(page: Page, cookies: Cookie[]): Promise<void>;++ /**+ * Gracefully closes the browser this controller owns. After this resolves,+ * the controller is no longer usable.+ */+ close(): Promise<void>;+}+export interface BrowserControllerEvents<Library extends CommonLibrary,LibraryOptions extends Dictionary | undefined = Parameters<Library['launch']>[0],@@ -38,7 +108,10 @@ export abstract class BrowserController<LaunchResult extends CommonBrowser = UnwrapPromise<ReturnType<Library['launch']>>,NewPageOptions = Parameters<LaunchResult['newPage']>[0],NewPageResult = UnwrapPromise<ReturnType<LaunchResult['newPage']>>,-> extends TypedEmitter<BrowserControllerEvents<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>> {+>+ extends TypedEmitter<BrowserControllerEvents<Library, LibraryOptions, LaunchResult, NewPageOptions, NewPageResult>>+ implements IBrowserController<NewPageResult>+{id = nanoid();protected log!: CrawleeLogger;diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.tsindex be9572b72..00a0c32c7 100644--- a/packages/browser-pool/src/browser-pool.ts+++ b/packages/browser-pool/src/browser-pool.ts@@ -1,6 +1,7 @@import { AsyncResource } from 'node:async_hooks';-import { type CrawleeLogger, serviceLocator } from '@crawlee/core';+import { type CrawleeLogger, SessionError, serviceLocator } from '@crawlee/core';+import type { IBrowserPool, NewPageOptions, PageState } from '@crawlee/types';import type { BrowserFingerprintWithHeaders } from 'fingerprint-generator';import { FingerprintGenerator } from 'fingerprint-generator';import { FingerprintInjector } from 'fingerprint-injector';@@ -24,6 +25,7 @@ import type { FingerprintGeneratorOptions } from './fingerprinting/types.js';import type { LaunchContext } from './launch-context.js';import type { InferBrowserPluginArray, UnwrapPromise } from './utils.js';+const PAGE_CLOSE_TIMEOUT_MILLIS = 5000;const PAGE_CLOSE_KILL_TIMEOUT_MILLIS = 1000;const BROWSER_KILLER_INTERVAL_MILLIS = 10 * 1000;@@ -302,7 +304,10 @@ export class BrowserPool<PageReturn extends UnwrapPromise<ReturnType<BrowserControllerReturn['newPage']>> = UnwrapPromise<ReturnType<BrowserControllerReturn['newPage']>>,-> extends TypedEmitter<BrowserPoolEvents<BrowserControllerReturn, PageReturn>> {+>+ extends TypedEmitter<BrowserPoolEvents<BrowserControllerReturn, PageReturn>>+ implements IBrowserPool<PageReturn>+{browserPlugins: BrowserPlugins;maxOpenPagesPerBrowser: number;retireBrowserAfterPageCount: number;@@ -436,14 +441,27 @@ export class BrowserPool<* Opens a new page in one of the running browsers or launches* a new browser and opens a page there, if no browsers are active,* or their page limits have been exceeded.+ *+ * **Session injection (best-effort):** When a {@apilink NewPageOptions.session|session} is+ * provided, this implementation uses it as a cache key for browser fingerprints (when+ * fingerprinting is enabled) and reads+ * {@apilink ProxyInfo.url|session.proxyInfo.url} /+ * {@apilink ProxyInfo.ignoreTlsErrors|session.proxyInfo.ignoreTlsErrors} as defaults+ * for `proxyUrl` and `ignoreTlsErrors` respectively. Explicit `proxyUrl` /+ * `ignoreTlsErrors` values in the options take precedence.+ *+ * Beyond fingerprint caching and proxy configuration, no other session+ * properties are consumed — cookie and header injection remain the+ * crawler's responsibility.*/async newPage(options: BrowserPoolNewPageOptions<PageOptions, BrowserPlugins[number]> = {}): Promise<PageReturn> {const {id = nanoid(),pageOptions,browserPlugin = this._pickBrowserPlugin(),- proxyUrl,- ignoreTlsErrors,+ session,+ proxyUrl = session?.proxyInfo?.url,+ ignoreTlsErrors = session?.proxyInfo?.ignoreTlsErrors,} = options;if (this.pages.has(id)) {@@ -659,6 +677,71 @@ export class BrowserPool<if (browserController) this.retireBrowserController(browserController);}+ /**+ * Releases a page back to the pool. The page is closed and, if the+ * optional `error` is a {@apilink SessionError}, the browser controller+ * that served the page is retired so that its tainted state (cookies,+ * storage, etc.) cannot leak into future sessions.+ *+ * This is the primary way the crawler should return pages to the pool.+ *+ * @param page The page to release.+ * @param options.error The error that caused the page to be released, if any.+ */+ async closePage(page: PageReturn, options?: { error?: Error }): Promise<void> {+ if (options?.error instanceof SessionError) {+ this.retireBrowserByPage(page);+ }++ // Puppeteer 25+ can hang `page.close()` indefinitely when the page's navigation was aborted, don't let it block the crawler.+ await addTimeoutToPromise(+ async () => page.close(),+ PAGE_CLOSE_TIMEOUT_MILLIS,+ `page.close() timed out after ${PAGE_CLOSE_TIMEOUT_MILLIS / 1000} seconds`,+ );+ }++ /**+ * Extracts the relevant state (currently just cookies) from a page via its+ * owning {@apilink BrowserController}. Returns empty state when the page is+ * no longer associated with a controller.+ *+ * As with {@apilink BrowserPool.injectPageState}, cookies are isolated per+ * page only when the pool is configured with `useIncognitoPages: true`.+ * With the default `useIncognitoPages: false`, the extracted cookies+ * include those set by any sibling page sharing the same browser.+ */+ async extractPageState(page: PageReturn): Promise<PageState> {+ const controller = this.getBrowserControllerByPage(page);++ if (!controller) {+ return { cookies: [] };+ }++ return { cookies: await controller.getCookies(page) };+ }++ /**+ * Injects state into a page via its owning {@apilink BrowserController}.+ *+ * No-op when the page is no longer associated with a controller.+ *+ * Note that cookies are isolated per page only when the pool is configured+ * with `useIncognitoPages: true` — each page then gets its own browser+ * context. With the default `useIncognitoPages: false`, all pages in a+ * browser share a single context, so injected cookies are visible to every+ * page served by that browser.+ */+ async injectPageState(page: PageReturn, state: PageState): Promise<void> {+ const controller = this.getBrowserControllerByPage(page);++ if (!controller) {+ return;+ }++ await controller.setCookies(page, state.cookies);+ }+/*** Removes all active browsers from the pool. The browsers will be* closed after all their pages are closed.@@ -889,12 +972,33 @@ export class BrowserPool<}}-export interface BrowserPoolNewPageOptions<PageOptions, BP extends BrowserPlugin> {+export interface BrowserPoolNewPageOptions<PageOptions, BP extends BrowserPlugin> extends NewPageOptions {/**- * Assign a custom ID to the page. If you don't a random string ID- * will be generated.+ * The proxy URL the pool uses internally to route the page: it keys browser+ * reuse (with `browserPerProxy`, only a browser already on this proxy is+ * reused), configures the launched browser, and is applied to incognito+ * pages. When omitted, it is derived from the+ * {@apilink NewPageOptions.session|session}'s `proxyInfo`; an explicit value+ * here takes precedence.+ *+ * This is an implementation detail of the built-in `BrowserPool`'s proxy+ * handling and is intentionally not part of the {@apilink IBrowserPool}+ * contract — through that interface the proxy is supplied via the session.*/- id?: string;+ proxyUrl?: string;+ /**+ * Disable TLS certificate verification for MITM proxies. Applied both when+ * launching a new browser and when creating a page in an existing one. When+ * omitted, it is derived from the+ * {@apilink NewPageOptions.session|session}'s `proxyInfo`; an explicit value+ * here takes precedence.+ *+ * This is an implementation detail of the built-in `BrowserPool` and is+ * intentionally not part of the {@apilink IBrowserPool} contract — through+ * that interface, configure it via the session's `proxyInfo` or through the+ * browser's `launchOptions`.+ */+ ignoreTlsErrors?: boolean;/*** Some libraries (Playwright) allow you to open new pages with specific* options. Use this property to set those options.@@ -909,15 +1013,6 @@ export interface BrowserPoolNewPageOptions<PageOptions, BP extends BrowserPlugin* see the `newPageInNewBrowser` function.*/browserPlugin?: BP;- /**- * Proxy URL.- */- proxyUrl?: string;- /**- * Disable TLS certificate verification for MITM proxies.- * Applied both when launching a new browser and when creating a page in an existing one.- */- ignoreTlsErrors?: boolean;}export interface BrowserPoolNewPageInNewBrowserOptions<PageOptions, BP extends BrowserPlugin> {diff --git a/packages/browser-pool/src/index.ts b/packages/browser-pool/src/index.tsindex 3b3ff37f6..f8502a42d 100644--- a/packages/browser-pool/src/index.ts+++ b/packages/browser-pool/src/index.ts@@ -33,7 +33,11 @@ export type {GetFingerprintReturn,} from './fingerprinting/types.js';export { BrowserName, DeviceCategory, OperatingSystemsName } from './fingerprinting/types.js';-export type { BrowserControllerEvents } from './abstract-classes/browser-controller.js';+export type {+ BrowserControllerEvents,+ IBrowserController,+ IBrowserLaunchContext,+} from './abstract-classes/browser-controller.js';export { BrowserController } from './abstract-classes/browser-controller.js';export { PuppeteerController } from './puppeteer/puppeteer-controller.js';export { PlaywrightController } from './playwright/playwright-controller.js';@@ -49,3 +53,4 @@ export type { LaunchContextOptions } from './launch-context.js';export { LaunchContext } from './launch-context.js';export type { InferBrowserPluginArray, UnwrapPromise } from './utils.js';export { anonymizeProxySugar, type AnonymizeProxySugarOptions } from './anonymize-proxy.js';+export type { IBrowserPool, NewPageOptions } from '@crawlee/types';diff --git a/packages/core/src/validators.ts b/packages/core/src/validators.tsindex da5c15d11..95095271a 100644--- a/packages/core/src/validators.ts+++ b/packages/core/src/validators.ts@@ -20,6 +20,11 @@ export const validators = {validator: ow.isValid(value, ow.object.hasKeys('fetchNextRequest', 'addRequest')),message: (label: string) => `Expected argument '${label}' to be a RequestQueue, got something else.`,}),+ browserPool: (value: Dictionary) => ({+ validator: ow.isValid(value, ow.object.hasKeys('newPage', 'closePage', 'extractPageState', 'injectPageState')),+ message: (label: string) =>+ `Expected argument '${label}' to implement the IBrowserPool interface (missing one of 'newPage', 'closePage', 'extractPageState', 'injectPageState'), got something else.`,+ }),sessionPool: (value: Dictionary) => ({validator: ow.isValid(value, ow.object.hasKeys('getSession')),message: (label: string) =>diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.tsindex f900adfd8..8c1f1bfe9 100644--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts@@ -10,7 +10,7 @@ import type {RoutesFromSchemas,} from '@crawlee/browser';import { BrowserCrawler, RequestState, Router, serviceLocator } from '@crawlee/browser';-import type { BrowserPoolOptions, PlaywrightController, PlaywrightPlugin } from '@crawlee/browser-pool';+import type { BrowserPoolOptions, PlaywrightPlugin } from '@crawlee/browser-pool';import type { Dictionary } from '@crawlee/types';import ow from 'ow';import type { Download, LaunchOptions, Page, Response } from 'playwright';@@ -32,9 +32,7 @@ import { gotoExtended, playwrightUtils } from './utils/playwright-utils.js';export type PlaywrightGotoOptions = NonNullable<Parameters<Page['goto']>[1]>;export interface PlaywrightCrawlingContext<UserData extends Dictionary = Dictionary>- extends- BrowserCrawlingContext<Page, Response, PlaywrightController, UserData, PlaywrightGotoOptions>,- PlaywrightContextUtils {}+ extends BrowserCrawlingContext<Page, Response, UserData, PlaywrightGotoOptions>, PlaywrightContextUtils {}export interface PlaywrightHook extends BrowserHook<PlaywrightCrawlingContext> {}export interface PlaywrightCrawlerOptions<@@ -43,7 +41,6 @@ export interface PlaywrightCrawlerOptions<> extends BrowserCrawlerOptions<Page,Response,- PlaywrightController,PlaywrightCrawlingContext,ContextExtension,ExtendedContext,@@ -61,8 +58,6 @@ export interface PlaywrightCrawlerOptions<* - `request` is an instance of the {@apilink Request} object with details about the URL to open, HTTP method etc.* - `page` is an instance of the `Playwright`* [`Page`](https://playwright.dev/docs/api/class-page)- * - `browserController` is an instance of the- * [`BrowserController`](https://github.com/apify/browser-pool#browsercontroller),* - `response` is an instance of the `Playwright`* [`Response`](https://playwright.dev/docs/api/class-response),* which is the main resource response as returned by `page.goto(request.url)`.@@ -188,7 +183,6 @@ export class PlaywrightCrawler<> extends BrowserCrawler<Page,Response,- PlaywrightController,{ browserPlugins: [PlaywrightPlugin] },LaunchOptions,PlaywrightCrawlingContext,@@ -256,7 +250,7 @@ export class PlaywrightCrawler<return gotoExtended(crawlingContext.page, crawlingContext.request, gotoOptions);}- private async enhanceContext(context: BrowserCrawlingContext<Page, Response, PlaywrightController, Dictionary>) {+ private async enhanceContext(context: BrowserCrawlingContext<Page, Response, Dictionary>) {const waitForSelector = async (selector: string, timeoutMs = 5_000) => {const locator = context.page.locator(selector).first();await locator.waitFor({ timeout: timeoutMs, state: 'attached' });diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.tsindex ea9b92a85..f846b0cbf 100644--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts@@ -9,7 +9,7 @@ import type {RoutesFromSchemas,} from '@crawlee/browser';import { BrowserCrawler, RequestState, Router } from '@crawlee/browser';-import type { BrowserPoolOptions, PuppeteerController, PuppeteerPlugin } from '@crawlee/browser-pool';+import type { BrowserPoolOptions, PuppeteerPlugin } from '@crawlee/browser-pool';import { serviceLocator } from '@crawlee/core';import type { Dictionary } from '@crawlee/types';import ow from 'ow';@@ -33,9 +33,7 @@ import { gotoExtended, puppeteerUtils } from './utils/puppeteer_utils.js';export type PuppeteerGoToOptions = NonNullable<Parameters<Page['goto']>[1]>;export interface PuppeteerCrawlingContext<UserData extends Dictionary = Dictionary>- extends- BrowserCrawlingContext<Page, HTTPResponse, PuppeteerController, UserData, PuppeteerGoToOptions>,- PuppeteerContextUtils {}+ extends BrowserCrawlingContext<Page, HTTPResponse, UserData, PuppeteerGoToOptions>, PuppeteerContextUtils {}export interface PuppeteerHook extends BrowserHook<PuppeteerCrawlingContext> {}export interface PuppeteerCrawlerOptions<@@ -44,7 +42,6 @@ export interface PuppeteerCrawlerOptions<> extends BrowserCrawlerOptions<Page,HTTPResponse,- PuppeteerController,PuppeteerCrawlingContext,ContextExtension,ExtendedContext,@@ -163,7 +160,6 @@ export class PuppeteerCrawler<> extends BrowserCrawler<Page,HTTPResponse,- PuppeteerController,{ browserPlugins: [PuppeteerPlugin] },LaunchOptions,PuppeteerCrawlingContext,@@ -219,7 +215,6 @@ export class PuppeteerCrawler<...(browserCrawlerOptions as BrowserCrawlerOptions<Page,HTTPResponse,- PuppeteerController,PuppeteerCrawlingContext,ContextExtension,ExtendedContext@@ -235,7 +230,7 @@ export class PuppeteerCrawler<return super.buildContextPipeline().compose({ action: this.enhanceContext.bind(this) });}- private async enhanceContext(context: BrowserCrawlingContext<Page, HTTPResponse, PuppeteerController>) {+ private async enhanceContext(context: BrowserCrawlingContext<Page, HTTPResponse>) {const waitForSelector = async (selector: string, timeoutMs = 5_000) => {await context.page.waitForSelector(selector, { timeout: timeoutMs });};diff --git a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts b/packages/stagehand-crawler/src/internals/stagehand-crawler.tsindex 17be591e7..3f18d3a2e 100644--- a/packages/stagehand-crawler/src/internals/stagehand-crawler.ts+++ b/packages/stagehand-crawler/src/internals/stagehand-crawler.ts@@ -218,7 +218,6 @@ export type StagehandGotoOptions = NonNullable<Parameters<Page['goto']>[1]>;export interface StagehandCrawlingContext<UserData extends Dictionary = Dictionary> extends BrowserCrawlingContext<StagehandPage,Response,- StagehandController,UserData,StagehandGotoOptions> {@@ -254,7 +253,6 @@ export interface StagehandCrawlerOptions<> extends BrowserCrawlerOptions<StagehandPage,Response,- StagehandController,StagehandCrawlingContext,ContextExtension,ExtendedContext,@@ -277,7 +275,6 @@ export interface StagehandCrawlerOptions<* The function receives the {@apilink StagehandCrawlingContext} as an argument, where:* - `request` is an instance of the {@apilink Request} object with details about the URL to open, HTTP method etc.* - `page` is an enhanced Playwright [`Page`](https://playwright.dev/docs/api/class-page) with AI methods- * - `browserController` is an instance of {@apilink StagehandController}* - `response` is the main resource response as returned by `page.goto(request.url)`* - `stagehand` is the Stagehand instance for advanced control*@@ -384,7 +381,6 @@ export class StagehandCrawler<> extends BrowserCrawler<StagehandPage,Response,- StagehandController,{ browserPlugins: [StagehandPlugin] },LaunchOptions,StagehandCrawlingContext,@@ -439,14 +435,42 @@ export class StagehandCrawler<return super.buildContextPipeline().compose({ action: this.setUpStagehand.bind(this) });}+ /**+ * Resolves the {@apilink StagehandController} that owns the given page, or+ * `undefined` when the pool does not expose controllers (e.g. a custom+ * {@apilink IBrowserPool} implementation).+ *+ * Stagehand needs direct controller access to reach the `Stagehand`+ * instance bound to the page's browser, which is why it reaches past the+ * {@apilink IBrowserPool} abstraction here.+ */+ private getBrowserControllerByPage(page: StagehandPage): StagehandController | undefined {+ if ('getBrowserControllerByPage' in this.browserPool) {+ return (+ this.browserPool as unknown as {+ getBrowserControllerByPage(page: StagehandPage): StagehandController | undefined;+ }+ ).getBrowserControllerByPage(page);+ }++ return undefined;+ }+/*** Enhance the page with Stagehand AI methods.*/private async setUpStagehand(crawlingContext: {- browserController: StagehandController;page: Page;}): Promise<{ stagehand: Stagehand; page: StagehandPage }> {- const stagehand = crawlingContext.browserController.getStagehand();+ const controller = this.getBrowserControllerByPage(crawlingContext.page as StagehandPage);++ if (!controller) {+ throw new Error(+ 'Could not resolve StagehandController for page — is the browser pool configured correctly?',+ );+ }++ const stagehand = controller.getStagehand();return {stagehand,diff --git a/packages/types/src/browser.ts b/packages/types/src/browser.tsindex b0227df86..7e39f9de8 100644--- a/packages/types/src/browser.ts+++ b/packages/types/src/browser.ts@@ -1,3 +1,4 @@+import type { ISession } from './session.js';import type { Dictionary } from './utility-types.js';export interface Cookie {@@ -62,3 +63,110 @@ export interface BrowserLikeResponse {url(): string;headers(): Dictionary<string | string[]>;}++/**+ * A snapshot of the relevant state of a page, as extracted by+ * {@apilink IBrowserPool.extractPageState}.+ */+export interface PageState {+ /**+ * Cookies currently set in the page's browsing context.+ */+ cookies: Cookie[];+}++/**+ * Options accepted by {@apilink IBrowserPool.newPage}.+ */+export interface NewPageOptions {+ /**+ * Assign a custom ID to the page. If you don't provide one, a random string+ * ID is generated.+ */+ id?: string;+ /**+ * The crawling session that will use the returned page.+ *+ * The pool derives proxy configuration from the session's {@apilink+ * ProxyInfo|proxy} (including TLS-error handling) — there are intentionally+ * no standalone `proxyUrl` / `ignoreTlsErrors` options; configure them+ * through the session instead.+ *+ * Session injection is **best-effort**: the pool may use the session's+ * {@apilink ProxyInfo|proxy}, cookies, or fingerprint data to configure the+ * page or the underlying browser, but none of this is guaranteed. Different+ * pool implementations (or pool configurations such as `useIncognitoPages`)+ * may support different subsets of session properties — or ignore them+ * entirely.+ *+ * The crawler is still responsible for deterministic session setup (e.g.+ * injecting cookies into the page before navigation) that must happen+ * regardless of pool implementation.+ */+ session?: ISession;+}++/**+ * Minimal contract that any object passed to a browser crawler as its `browserPool`+ * option must satisfy.+ *+ * Lifecycle (`destroy`) is the responsibility of whoever owns the pool — since a+ * user-supplied pool is never owned by the crawler, the crawler never tears it+ * down.+ *+ * Implement this interface to plug a custom page-provisioning strategy into any+ * Crawlee browser crawler — for example a remote browser farm, a session-aware+ * pool that pins pages to fingerprints differently, or a thin wrapper around the+ * built-in `BrowserPool`.+ *+ * @category Browser management+ */+export interface IBrowserPool<Page = unknown> {+ /**+ * Opens a new page. The pool decides which browser to use, launching a new+ * one if needed.+ */+ newPage(options?: NewPageOptions): Promise<Page>;++ /**+ * Signals the pool that the caller is done with the page. The pool is+ * responsible for closing the page and performing any necessary cleanup+ * (e.g. retiring the underlying browser when a session has gone bad).+ *+ * @param page The page to release back to the pool.+ * @param options.error If the page is being released because of an error,+ * pass the error here. In particular, if the error is a+ * {@apilink SessionError}, implementations should treat it as a signal+ * to purge all state associated with the session (e.g. discard any+ * browser that served the page).+ */+ closePage(page: Page, options?: { error?: Error }): Promise<void>;++ /**+ * Extracts the relevant state from a page so the caller can persist it —+ * for example, back-propagating cookies into the crawling {@apilink+ * ISession|session}.+ *+ * @param page The page to read state from.+ */+ extractPageState(page: Page): Promise<PageState>;++ /**+ * Injects state (currently just cookies) into a page. This is the+ * counterpart to {@apilink IBrowserPool.extractPageState} and lets the+ * caller set up a page — for example, seeding it with the crawling+ * {@apilink ISession|session}'s cookies before navigation.+ *+ * As with {@apilink IBrowserPool.newPage}, the caller decides *what* state+ * to inject, while the pool decides *how*.+ *+ * Isolation between pages is **best-effort**: depending on the pool+ * implementation and its configuration, multiple pages may share a browsing+ * context, so injected state (such as cookies) can bleed across pages+ * served by the same underlying browser.+ *+ * @param page The page to inject state into.+ * @param state The state to inject.+ */+ injectPageState(page: Page, state: PageState): Promise<void>;+}diff --git a/test/core/crawlers/basic_browser_crawler.ts b/test/core/crawlers/basic_browser_crawler.tsindex 83c5f80a3..6b0ff9778 100644--- a/test/core/crawlers/basic_browser_crawler.ts+++ b/test/core/crawlers/basic_browser_crawler.ts@@ -1,4 +1,4 @@-import type { PuppeteerController, PuppeteerPlugin } from '@crawlee/browser-pool';+import type { PuppeteerPlugin } from '@crawlee/browser-pool';import type {BrowserCrawlerOptions,BrowserCrawlingContext,@@ -6,22 +6,20 @@ import type {PuppeteerGoToOptions,} from '@crawlee/puppeteer';import { BrowserCrawler } from '@crawlee/puppeteer';+import type { Dictionary } from '@crawlee/types';// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll goooooodimport type { HTTPResponse, LaunchOptions, Page } from 'puppeteer';-export type TestCrawlingContext = BrowserCrawlingContext<Page, HTTPResponse, PuppeteerController>;+export type TestCrawlingContext = BrowserCrawlingContext<Page, HTTPResponse, Dictionary>;export class BrowserCrawlerTest extends BrowserCrawler<Page,HTTPResponse,- PuppeteerController,{ browserPlugins: [PuppeteerPlugin] },LaunchOptions,TestCrawlingContext> {- constructor(- options: Partial<BrowserCrawlerOptions<Page, HTTPResponse, PuppeteerController, TestCrawlingContext>> = {},- ) {+ constructor(options: Partial<BrowserCrawlerOptions<Page, HTTPResponse, TestCrawlingContext>> = {}) {super({...options,contextPipelineBuilder: () => this.buildContextPipeline(),diff --git a/test/core/crawlers/browser_crawler.test.ts b/test/core/crawlers/browser_crawler.test.tsindex 6b48d7d0b..2042b6441 100644--- a/test/core/crawlers/browser_crawler.test.ts+++ b/test/core/crawlers/browser_crawler.test.ts@@ -1,6 +1,12 @@import type { Server } from 'node:http';-import { BROWSER_POOL_EVENTS, OperatingSystemsName, PuppeteerPlugin } from '@crawlee/browser-pool';+import type { BrowserPool, PuppeteerController } from '@crawlee/browser-pool';+import {+ BROWSER_POOL_EVENTS,+ BrowserPool as BrowserPoolClass,+ OperatingSystemsName,+ PuppeteerPlugin,+} from '@crawlee/browser-pool';import { bindMethodsToServiceLocator, BLOCKED_STATUS_CODES, ServiceLocator, SessionPool } from '@crawlee/core';import type { PuppeteerGoToOptions } from '@crawlee/puppeteer';import { EnqueueStrategy, ProxyConfiguration, Request, RequestList, RequestState, Session } from '@crawlee/puppeteer';@@ -124,8 +130,9 @@ describe('BrowserCrawler', () => {// Spy on destroy and track if it was calledlet destroyCalled = false;- const originalDestroy = browserCrawler.browserPool.destroy.bind(browserCrawler.browserPool);- browserCrawler.browserPool.destroy = async () => {+ const ownedPool = browserCrawler.browserPool as BrowserPool;+ const originalDestroy = ownedPool.destroy.bind(ownedPool);+ ownedPool.destroy = async () => {destroyCalled = true;return originalDestroy();};@@ -137,6 +144,40 @@ describe('BrowserCrawler', () => {}});+ test.concurrent('should not tear down a user-supplied browser pool', async () => {+ const localStorageEmulator = new MemoryStorageEmulator();+ await localStorageEmulator.init();+ const puppeteerPlugin = new PuppeteerPlugin(puppeteer);+ const externalPool = new BrowserPoolClass({ browserPlugins: [puppeteerPlugin] });++ try {+ const requestList = await RequestList.open({+ sources: [{ url: 'http://example.com/?q=1' }],+ });+ const browserCrawler = new BrowserCrawlerTest({+ browserPool: externalPool,+ requestList,+ requestHandler: async () => {},+ maxRequestRetries: 1,+ });++ expect(browserCrawler.browserPool).toBe(externalPool);++ let destroyCalled = false;+ const originalDestroy = externalPool.destroy.bind(externalPool);+ externalPool.destroy = async () => {+ destroyCalled = true;+ return originalDestroy();+ };++ await browserCrawler.run();+ expect(destroyCalled).toBe(false);+ } finally {+ await externalPool.destroy();+ await localStorageEmulator.destroy();+ }+ });+test.concurrent('should retire session after TimeoutError', async () => {const localStorageEmulator = new MemoryStorageEmulator();await localStorageEmulator.init();@@ -852,7 +893,7 @@ describe('BrowserCrawler', () => {},maxRequestRetries: 1,});- browserCrawler.browserPool.on(BROWSER_POOL_EVENTS.BROWSER_RETIRED, () => {+ (browserCrawler.browserPool as BrowserPool).on(BROWSER_POOL_EVENTS.BROWSER_RETIRED, () => {retiredBrowserCount += 1;});@@ -904,30 +945,34 @@ describe('BrowserCrawler', () => {await localStorageEmulator.init();const puppeteerPlugin = new PuppeteerPlugin(puppeteer);+ const pool = new BrowserPoolClass({+ browserPlugins: [puppeteerPlugin],+ useFingerprints: true,+ fingerprintOptions: {+ fingerprintGeneratorOptions: {+ operatingSystems: [OperatingSystemsName.windows],+ },+ },+ });+try {const requestList = await RequestList.open({sources: [{ url: `${serverAddress}/?q=1` }],});const browserCrawler = new BrowserCrawlerTest({- browserPoolOptions: {- browserPlugins: [puppeteerPlugin],- useFingerprints: true,- fingerprintOptions: {- fingerprintGeneratorOptions: {- operatingSystems: [OperatingSystemsName.windows],- },- },- },+ browserPool: pool,requestList,- requestHandler: async ({ browserController }) => {- expect(browserController.launchContext.fingerprint).toBeDefined();+ requestHandler: async ({ page }) => {+ const controller = pool.getBrowserControllerByPage(page);+ expect(controller?.launchContext.fingerprint).toBeDefined();},});await browserCrawler.run();expect.hasAssertions();} finally {+ await pool.destroy();await localStorageEmulator.destroy();}});@@ -969,8 +1014,8 @@ describe('BrowserCrawler', () => {maxConcurrency: 1,});- browserCrawler.browserPool.postLaunchHooks.push((_pageId, browserController) => {- browserProxies.push(browserController.launchContext.proxyUrl!);+ (browserCrawler.browserPool as BrowserPool).postLaunchHooks.push((_pageId, browserController) => {+ browserProxies.push((browserController as PuppeteerController).launchContext.proxyUrl!);});await browserCrawler.run();diff --git a/test/e2e/stagehand-concurrent/actor/main.js b/test/e2e/stagehand-concurrent/actor/main.jsindex 5b3a6e899..95ee351c5 100644--- a/test/e2e/stagehand-concurrent/actor/main.js+++ b/test/e2e/stagehand-concurrent/actor/main.js@@ -25,11 +25,12 @@ await Actor.main(async () => {model: 'anthropic/claude-haiku-4-5-20251001',verbose: 0,},- async requestHandler({ page, request, browserController, log, pushData }) {+ async requestHandler({ page, request, log, pushData }) {log.info(`Processing ${request.loadedUrl}`);- // Track which browser instance handled this request- const browserId = browserController.id;+ // Track which browser instance handled this request via the underlying Playwright browser+ const browser = page.context().browser();+ const browserId = browser?.process()?.pid ?? 'unknown';browserIds.add(browserId);// Simple extraction - just get the page title