Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: drop the `closeCookieModals` helpers (#4073)

Jindřich Bär0369fed44c4b
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index c7c33ba67..8aac6b5d7 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -146,9 +146,6 @@ interface BlockRequestsOptions {
// @public (undocumented)
type ClickOptions = Parameters<Page['click']>[1];
-// @public (undocumented)
-function closeCookieModals(page: Page): Promise<void>;
-
// @public (undocumented)
type CompiledScriptFunction = (params: CompiledScriptParams) => Promise<unknown>;
@@ -296,7 +293,6 @@ declare namespace playwrightClickElements {
// @public (undocumented)
interface PlaywrightContextUtils {
blockRequests(options?: BlockRequestsOptions): Promise<void>;
- closeCookieModals(): Promise<void>;
compileScript(scriptString: string, ctx?: Dictionary): CompiledScriptFunction;
enqueueLinksByClickingElements(options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>): Promise<BatchAddRequestsResult>;
handleCloudflareChallenge(options?: HandleCloudflareChallengeOptions): Promise<Response_2 | undefined>;
@@ -364,7 +360,6 @@ declare namespace playwrightUtils {
infiniteScroll,
saveSnapshot,
parseWithCheerio,
- closeCookieModals,
InjectFileOptions,
BlockRequestsOptions,
PlaywrightDirectNavigationOptions as DirectNavigationOptions,
diff --git a/docs/public-api/crawlee-puppeteer.api.md b/docs/public-api/crawlee-puppeteer.api.md
index 857ab7669..725adc754 100644
--- a/docs/public-api/crawlee-puppeteer.api.md
+++ b/docs/public-api/crawlee-puppeteer.api.md
@@ -58,9 +58,6 @@ const blockResources: (page: Page, resourceTypes?: string[]) => Promise<void>;
// @public @deprecated
function cacheResponses(page: Page, cache: Dictionary<Partial<ResponseForRequest>>, responseUrlRules: (string | RegExp)[]): Promise<void>;
-// @public (undocumented)
-function closeCookieModals(page: Page): Promise<void>;
-
// @public (undocumented)
export type CompiledScriptFunction = (params: CompiledScriptParams) => Promise<unknown>;
@@ -175,7 +172,6 @@ declare namespace puppeteerClickElements {
interface PuppeteerContextUtils {
addInterceptRequestHandler(handler: InterceptHandler): Promise<void>;
blockRequests(options?: BlockRequestsOptions): Promise<void>;
- closeCookieModals(): Promise<void>;
compileScript(scriptString: string, ctx?: Dictionary): CompiledScriptFunction;
enqueueLinksByClickingElements(options: Omit<EnqueueLinksByClickingElementsOptions, 'page' | 'requestManager'>): Promise<BatchAddRequestsResult>;
infiniteScroll(options?: InfiniteScrollOptions): Promise<void>;
@@ -250,7 +246,6 @@ declare namespace puppeteerUtils {
gotoExtended,
infiniteScroll,
saveSnapshot,
- closeCookieModals,
PuppeteerDirectNavigationOptions as DirectNavigationOptions,
InjectFileOptions,
BlockRequestsOptions,
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index c4c60067d..b7312d3e5 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -556,6 +556,10 @@ const crawler = new CheerioCrawler({
`extendContext` runs **before navigation**, so the members it returns are visible to the `preNavigationHooks`, `postNavigationHooks`, and the `requestHandler` alike. As a consequence, the `context` passed to `extendContext` is the pre-navigation context and does **not** include navigation-dependent members (e.g. `page`, `response`, `$`, `body`). If your extension needs to read those, do it in a `postNavigationHook` or the `requestHandler` instead.
+#### Crawling context no longer includes `closeCookieModals`
+
+The `closeCookieModals` context helper is removed from the Playwright and Puppeteer crawlers, along with the `playwrightUtils.closeCookieModals` / `puppeteerUtils.closeCookieModals` functions and the optional `idcac-playwright` peer dependency they were built on.
+
### Crawling context is strictly typed
Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
@@ -2123,6 +2127,7 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla
- `FileDownloadOptions.streamHandler` - streaming should now be handled directly in the `requestHandler` instead
- `playwrightUtils.registerUtilsToContext` and `puppeteerUtils.registerUtilsToContext` - this is now added to the context via `ContextPipeline` composition
- `context.blockResources` and `context.cacheResponses` — no longer attached to the crawling context. The functionality is still available as deprecated functions, accessible both via the `puppeteerUtils` namespace (`puppeteerUtils.blockResources`, `puppeteerUtils.cacheResponses`) and as top-level exports from `@crawlee/puppeteer` (`import { blockResources, cacheResponses } from '@crawlee/puppeteer'`). Unlike the old context helpers, these take an explicit `page` argument — e.g. `await blockResources(page)`. Both are `@deprecated` and will be removed in a future release, so migrate away from them.
+- `context.closeCookieModals`, `playwrightUtils.closeCookieModals` and `puppeteerUtils.closeCookieModals` — removed along with the optional `idcac-playwright` peer dependency (see [Crawling context no longer includes `closeCookieModals`](#crawling-context-no-longer-includes-closecookiemodals))
- `Configuration.systemInfoV2` / `CRAWLEE_SYSTEM_INFO_V2` environment variable — the v2 behavior is now the default (see [Available resource detection](#available-resource-detection))
- `checkAndSerialize` and `chunkBySize` functions (from `@crawlee/core`) — value (de)serialization now lives in the `KeyValueStore` frontend; use `serializeValue` / `parseValue` (see [`maybeStringify` is removed](#maybestringify-is-removed))
- `BASIC_CRAWLER_TIMEOUT_BUFFER_SECS` constant (from `@crawlee/basic`) — was an internal timeout buffer, no longer exported
diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
index 11ad028cf..90aeebaf1 100644
--- a/packages/crawlee/package.json
+++ b/packages/crawlee/package.json
@@ -66,14 +66,10 @@
"tslib": "^2.8.1"
},
"peerDependencies": {
- "idcac-playwright": "*",
"playwright": "*",
"puppeteer": "*"
},
"peerDependenciesMeta": {
- "idcac-playwright": {
- "optional": true
- },
"playwright": {
"optional": true
},
diff --git a/packages/playwright-crawler/package.json b/packages/playwright-crawler/package.json
index b1332db20..36c1798be 100644
--- a/packages/playwright-crawler/package.json
+++ b/packages/playwright-crawler/package.json
@@ -66,13 +66,9 @@
"zod": "catalog:"
},
"peerDependencies": {
- "idcac-playwright": "^0.2.0",
"playwright": "*"
},
"peerDependenciesMeta": {
- "idcac-playwright": {
- "optional": true
- },
"playwright": {
"optional": true
}
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index 0ccd7209b..1f35c39d4 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -334,7 +334,6 @@ export class PlaywrightCrawler<
requestManager: this.requestManager!,
}),
compileScript: (scriptString: string, ctx?: Dictionary) => playwrightUtils.compileScript(scriptString, ctx),
- closeCookieModals: async () => playwrightUtils.closeCookieModals(context.page),
handleCloudflareChallenge: async (options?: HandleCloudflareChallengeOptions) => {
return playwrightUtils.handleCloudflareChallenge(context.page, context.request.url, options);
},
diff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
index bc7fcb903..8e95548c2 100644
--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
@@ -632,37 +632,6 @@ export async function parseWithCheerio(
return $;
}
-let idcacPlaywright: null | { getInjectableScript: () => string } = null;
-async function getIdcacPlaywright() {
- if (idcacPlaywright) return idcacPlaywright;
-
- try {
- idcacPlaywright = await import('idcac-playwright');
- } catch (error: any) {
- getLog().warning(`Failed to import 'idcac-playwright'.
-
-We recently made idcac-playwright an optional dependency due to licensing issues.
-To use this feature, please install it manually by running
-
-npm install idcac-playwright
-
-Original error message follows:
-
-${error.message}
-`);
- }
- return idcacPlaywright;
-}
-
-export async function closeCookieModals(page: Page): Promise<void> {
- parseArgument(page, validators.browserPage);
- const idcac = await getIdcacPlaywright();
-
- if (idcac?.getInjectableScript()) {
- await page.evaluate(idcac.getInjectableScript());
- }
-}
-
export interface HandleCloudflareChallengeOptions {
/** Logging defaults to the `debug` level, use this flag to log to `info` level instead. */
verbose?: boolean;
@@ -1006,20 +975,6 @@ export interface PlaywrightContextUtils {
*/
compileScript(scriptString: string, ctx?: Dictionary): CompiledScriptFunction;
- /**
- * Tries to close cookie consent modals on the page. Based on the I Don't Care About Cookies browser extension.
- *
- * Note that this method requires the idcac-playwright package to be installed.
- * Crawlee does not include it by default due to licensing issues.
- *
- * To use this method, please install the package manually by running:
- *
- * ```bash
- * npm install idcac-playwright
- * ```
- */
- closeCookieModals(): Promise<void>;
-
/**
* This helper tries to solve the Cloudflare challenge automatically by clicking on the checkbox.
* It will try to detect the Cloudflare page, click on the checkbox, and wait for 10 seconds (configurable
@@ -1080,7 +1035,6 @@ export const playwrightUtils = {
infiniteScroll,
saveSnapshot,
compileScript,
- closeCookieModals,
RenderingTypePredictor,
handleCloudflareChallenge,
};
diff --git a/packages/puppeteer-crawler/package.json b/packages/puppeteer-crawler/package.json
index f0be0461a..3e14f3472 100644
--- a/packages/puppeteer-crawler/package.json
+++ b/packages/puppeteer-crawler/package.json
@@ -60,13 +60,9 @@
"zod": "catalog:"
},
"peerDependencies": {
- "idcac-playwright": "^0.2.0",
"puppeteer": "*"
},
"peerDependenciesMeta": {
- "idcac-playwright": {
- "optional": true
- },
"puppeteer": {
"optional": true
}
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index 6b1b39f70..7d6292823 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -314,7 +314,6 @@ export class PuppeteerCrawler<
...options,
configuration: serviceLocator.getConfiguration(),
}),
- closeCookieModals: async () => puppeteerUtils.closeCookieModals(context.page),
};
}
diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
index a69dc7ad5..dfca18387 100644
--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
@@ -778,37 +778,6 @@ export async function saveSnapshot(page: Page, options: SaveSnapshotOptions = {}
}
}
-let idcacPlaywright: null | { getInjectableScript: () => string } = null;
-async function getIdcacPlaywright() {
- if (idcacPlaywright) return idcacPlaywright;
-
- try {
- idcacPlaywright = await import('idcac-playwright');
- } catch (error: any) {
- getLog().warning(`Failed to import 'idcac-playwright'.
-
-We recently made idcac-playwright an optional dependency due to licensing issues.
-To use this feature, please install it manually by running
-
-npm install idcac-playwright
-
-Original error message follows:
-
-${error.message}
-`);
- }
- return idcacPlaywright;
-}
-
-export async function closeCookieModals(page: Page): Promise<void> {
- parseArgument(page, validators.browserPage);
- const idcac = await getIdcacPlaywright();
-
- if (idcac?.getInjectableScript()) {
- await page.evaluate(idcac.getInjectableScript());
- }
-}
-
export interface PuppeteerContextUtils {
/**
* Injects a JavaScript file into current `page`.
@@ -1053,20 +1022,6 @@ export interface PuppeteerContextUtils {
* Saves a full screenshot and HTML of the current page into a Key-Value store.
*/
saveSnapshot(options?: SaveSnapshotOptions): Promise<void>;
-
- /**
- * Tries to close cookie consent modals on the page. Based on the I Don't Care About Cookies browser extension.
- *
- * Note that this method requires the idcac-playwright package to be installed.
- * Crawlee does not include it by default due to licensing issues.
- *
- * To use this method, please install the package manually by running:
- *
- * ```bash
- * npm install idcac-playwright
- * ```
- */
- closeCookieModals(): Promise<void>;
}
export { enqueueLinksByClickingElements, addInterceptRequestHandler, removeInterceptRequestHandler };
@@ -1084,5 +1039,4 @@ export const puppeteerUtils = {
infiniteScroll,
saveSnapshot,
parseWithCheerio,
- closeCookieModals,
};
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index af3f708fd..8bdcd9e02 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -790,9 +790,6 @@ importers:
'@crawlee/utils':
specifier: workspace:*
version: link:../utils
- idcac-playwright:
- specifier: '*'
- version: 0.2.0
import-local:
specifier: ^3.2.0
version: 3.2.0
@@ -997,9 +994,6 @@ importers:
cheerio:
specifier: ^1.0.0
version: 1.2.0
- idcac-playwright:
- specifier: ^0.2.0
- version: 0.2.0
jquery:
specifier: ^3.7.1
version: 3.7.1
@@ -1051,9 +1045,6 @@ importers:
devtools-protocol:
specifier: '*'
version: 0.0.1666840
- idcac-playwright:
- specifier: ^0.2.0
- version: 0.2.0
jquery:
specifier: ^3.7.1
version: 3.7.1
@@ -8330,9 +8321,6 @@ packages:
peerDependencies:
postcss: ^8.1.0
- resolution: {integrity: sha512-qJH7vQgq3TKnhea/3Z3jlEJL7NC9vK9BkLClAzQHVRepBtq1fWfSI4fSuMKcPq7nDUTTlIEIS+vU+GRwwR1BXw==}
-
resolution: {integrity: sha512-ZrYyM0sozNPZlvBvE7Oq9Bn44n0qKGrYu5sQ0JzMUnjIhpgWYE2JB6aBoFwEYdPjqj7jPyxXTMJiHDOxDfd8yw==}
engines: {node: '>=18'}
@@ -21961,8 +21949,6 @@ snapshots:
dependencies:
postcss: 8.5.26
-
dependencies:
reserved-identifiers: 1.2.0
diff --git a/test/e2e/playwright-enqueue-links/actor/main.js b/test/e2e/playwright-enqueue-links/actor/main.js
index 221b413b8..ea500d77f 100644
--- a/test/e2e/playwright-enqueue-links/actor/main.js
+++ b/test/e2e/playwright-enqueue-links/actor/main.js
@@ -14,14 +14,12 @@ const mainOptions = {
await Actor.main(async () => {
const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 30,
- requestHandler: async ({ page, request, enqueueLinks, closeCookieModals }) => {
+ requestHandler: async ({ page, request, enqueueLinks }) => {
const { url, loadedUrl } = request;
const pageTitle = await page.title();
log.info(`URL: ${url}; LOADED_URL: ${loadedUrl}; TITLE: ${pageTitle}`);
- await closeCookieModals();
-
const results = await enqueueLinks();
if (loadedUrl.startsWith('https://drive')) {
diff --git a/test/e2e/playwright-enqueue-links/actor/package.json b/test/e2e/playwright-enqueue-links/actor/package.json
index ce445f0f5..197d3d7ef 100644
--- a/test/e2e/playwright-enqueue-links/actor/package.json
+++ b/test/e2e/playwright-enqueue-links/actor/package.json
@@ -11,7 +11,6 @@
"@crawlee/playwright": "file:./packages/playwright-crawler",
"@crawlee/types": "file:./packages/types",
"@crawlee/utils": "file:./packages/utils",
- "idcac-playwright": "^0.2.0",
"playwright": "1.58.2"
},
"overrides": {
diff --git a/test/e2e/puppeteer-enqueue-links/actor/main.js b/test/e2e/puppeteer-enqueue-links/actor/main.js
index 48415382f..226e5e70f 100644
--- a/test/e2e/puppeteer-enqueue-links/actor/main.js
+++ b/test/e2e/puppeteer-enqueue-links/actor/main.js
@@ -12,14 +12,12 @@ const mainOptions = {
await Actor.main(async () => {
const crawler = new PuppeteerCrawler({
maxRequestsPerCrawl: 30,
- async requestHandler({ page, enqueueLinks, request, log, closeCookieModals }) {
+ async requestHandler({ page, enqueueLinks, request, log }) {
const { url, loadedUrl } = request;
const pageTitle = await page.title();
log.info(`URL: ${url}; LOADED_URL: ${loadedUrl}; TITLE: ${pageTitle}`);
- await closeCookieModals();
-
const results = await enqueueLinks();
if (loadedUrl.startsWith('https://drive')) {
diff --git a/test/e2e/puppeteer-enqueue-links/actor/package.json b/test/e2e/puppeteer-enqueue-links/actor/package.json
index f058749f4..e683d0b38 100644
--- a/test/e2e/puppeteer-enqueue-links/actor/package.json
+++ b/test/e2e/puppeteer-enqueue-links/actor/package.json
@@ -11,7 +11,6 @@
"@crawlee/puppeteer": "file:./packages/puppeteer-crawler",
"@crawlee/types": "file:./packages/types",
"@crawlee/utils": "file:./packages/utils",
- "idcac-playwright": "^0.2.0",
"puppeteer": "*"
},
"overrides": {