Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(puppeteer-crawler): migrate cookie handling to browser-context API (#3572)

Martin Adámek0174b814eed5
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index 157071431..c61c28501 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -266,3 +266,17 @@ The `transformRequestFunction` callback receives a `RequestOptions` object and c
- A new `RequestOptions` plain object
- `'unchanged'` to keep the original options as-is
- A falsy value or `'skip'` to exclude the request from the queue
+
+## Puppeteer cookies are now read and written at the browser-context level
+
+The `PuppeteerController._getCookies` / `_setCookies` methods (used internally by the session pool to sync cookies between a `Session` and a Puppeteer page) now call `page.browserContext().cookies()` / `setCookie()` instead of the deprecated `page.cookies()` / `page.setCookie()`. The page-level API was removed in newer Puppeteer releases.
+
+This aligns the Puppeteer controller with the Playwright controller, which has always worked at the context level.
+
+**What changes in practice**
+- Cookie reads return every cookie stored in the page's browser context, not just cookies matching the page's current URL. If your `Session` relied on the URL-scoped filtering (for example, to avoid pulling cookies that belong to other tabs in the same context), you'll now see the full set.
+- Cookie writes are applied to the whole browser context. When you launch pages with shared contexts, cookies written via `Session.setCookiesFromResponse` or similar will be visible to every other page in that context.
+
+If you rely on Crawlee's default configuration (one browser context per session, which is the `useIncognitoPages` / `newContextPerSession` behavior used by `PuppeteerCrawler`), you should not notice any difference — each session already owns its own context.
+
+**Cookie `url` field** — the old `page.setCookie()` auto-filled a missing `url` on each cookie with the page's current URL. The new `browserContext().setCookie()` does not; Chromium rejects cookies that carry neither `url` nor `domain`. Crawlee's internal `_setCookies` keeps the old behavior by back-filling `page.url()` for any cookie that has neither field set, but if you call `browserContext().setCookie()` directly (outside of Crawlee) you need to provide one of them yourself.
diff --git a/packages/browser-pool/src/puppeteer/puppeteer-controller.ts b/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
index f1e482a18..7a888529f 100644
--- a/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
+++ b/packages/browser-pool/src/puppeteer/puppeteer-controller.ts
@@ -142,7 +142,7 @@ export class PuppeteerController extends BrowserController<
}
protected async _getCookies(page: PuppeteerTypes.Page): Promise<Cookie[]> {
- const cookies = await page.cookies();
+ const cookies = await page.browserContext().cookies();
// Puppeteer 25+ can report `sameSite: 'Default'`, which means the attribute is unspecified.
return cookies.map(({ sameSite, ...rest }) => ({
...rest,
@@ -151,6 +151,12 @@ export class PuppeteerController extends BrowserController<
}
protected async _setCookies(page: PuppeteerTypes.Page, cookies: Cookie[]): Promise<void> {
- return page.setCookie(...cookies);
+ // BrowserContext.setCookie requires `url` or `domain`; the page-level API used to back-fill
+ // the page's current URL for us. Replicate that so callers who pass neither don't get rejected.
+ const pageUrl = page.url();
+ const normalized = cookies.map((cookie) =>
+ cookie.url || cookie.domain ? cookie : { ...cookie, url: pageUrl },
+ );
+ return page.browserContext().setCookie(...(normalized as PuppeteerTypes.CookieData[]));
}
}