Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: mention `pageOptions` limitation (#1674)
Szymon Marczak0bc147e112e4
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.tsindex d00f0aa75..ebc25ec2a 100644--- a/packages/browser-crawler/src/internals/browser-crawler.ts+++ b/packages/browser-crawler/src/internals/browser-crawler.ts@@ -205,6 +205,9 @@ export interface BrowserCrawlerOptions<* },* ]* ```+ *+ * Modyfing `pageOptions` is supported only in Playwright incognito.+ * See {@apilink PrePageCreateHook}*/preNavigationHooks?: BrowserHook<Context>[];diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.tsindex b2e714a20..61946b56e 100644--- a/packages/browser-pool/src/browser-pool.ts+++ b/packages/browser-pool/src/browser-pool.ts@@ -108,7 +108,7 @@ export type PostLaunchHook<BC extends BrowserController> = (pageId: string, brow* The hooks are called with three arguments:* `pageId`: `string`, `browserController`: {@apilink BrowserController} and* `pageOptions`: `object|undefined` - This only works if the underlying `BrowserController` supports new page options.- * So far, new page options are only supported by `PlaywrightController`.+ * So far, new page options are only supported by `PlaywrightController` in incognito contexts.* If the page options are not supported by `BrowserController` the `pageOptions` argument is `undefined`.*/export type PrePageCreateHook<@@ -176,7 +176,7 @@ export interface BrowserPoolHooks<* The hooks are called with three arguments:* `pageId`: `string`, `browserController`: {@apilink BrowserController} and* `pageOptions`: `object|undefined` - This only works if the underlying `BrowserController` supports new page options.- * So far, new page options are only supported by `PlaywrightController`.+ * So far, new page options are only supported by `PlaywrightController` in incognito contexts.* If the page options are not supported by `BrowserController` the `pageOptions` argument is `undefined`.*/prePageCreateHooks?: PrePageCreateHook<BC>[];diff --git a/packages/http-crawler/src/internals/http-crawler.ts b/packages/http-crawler/src/internals/http-crawler.tsindex eb0af16c0..81d30b6a6 100644--- a/packages/http-crawler/src/internals/http-crawler.ts+++ b/packages/http-crawler/src/internals/http-crawler.ts@@ -91,6 +91,9 @@ export interface HttpCrawlerOptions<Context extends InternalHttpCrawlingContext* },* ]* ```+ *+ * Modyfing `pageOptions` is supported only in Playwright incognito.+ * See {@apilink PrePageCreateHook}*/preNavigationHooks?: InternalHttpHook<Context>[];diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.tsindex aaa676993..50a6e22f4 100644--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts@@ -92,6 +92,9 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<* },* ]* ```+ *+ * Modyfing `pageOptions` is supported only in Playwright incognito.+ * See {@apilink PrePageCreateHook}*/preNavigationHooks?: PlaywrightHook[];diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.tsindex c861afdad..b15c8716b 100644--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts@@ -42,6 +42,9 @@ export interface PuppeteerCrawlerOptions extends BrowserCrawlerOptions<* },* ]* ```+ *+ * Modyfing `pageOptions` is supported only in Playwright incognito.+ * See {@apilink PrePageCreateHook}*/preNavigationHooks?: PuppeteerHook[];