Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fix(core): log a warning on malformed cookie fragments in `mergeCookies` (#3850)
Jindřich Bär46313998b820
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex adc0391c0..a84fdcad1 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -739,6 +739,8 @@ const response = await sendRequest({ url: '...' }, { cookieJar: jar });The protected `HttpCrawler._applyCookies` method is removed. If you were overriding it in a subclass, move your logic to a `preNavigationHook` that sets cookies on `request.headers.Cookie` or on the `session` cookie jar directly.+`mergeCookies` now skips malformed cookie fragments with a warning instead of throwing.+### `Session.getCookies`, `setCookies` and `setCookiesFromResponse` are removedThe public cookie helper methods on `Session` — `getCookies(url)`, `setCookies(cookies, url)`, and `setCookiesFromResponse(response)` — have been removed as part of centralizing cookie assembly in `BaseHttpClient`. Work with the session's `cookieJar` directly, or use the new `Session.getCookieString(url)` to read the assembled `Cookie` header value.diff --git a/packages/core/src/cookie_utils.ts b/packages/core/src/cookie_utils.tsindex 53aeb9130..4213f9f19 100644--- a/packages/core/src/cookie_utils.ts+++ b/packages/core/src/cookie_utils.ts@@ -120,7 +120,10 @@ export function mergeCookies(url: string, sourceCookies: string[]): string {if (!cookieString) continue;const cookie = Cookie.parse(cookieString);- if (!cookie) throw new CookieParseError(cookieString);+ if (!cookie) {+ serviceLocator.getLogger().warning(`Skipping malformed cookie fragment: '${cookieString}'`);+ continue;+ }const similarKeyCookie = jar.getCookiesSync(url).find((c) => {return cookie.key !== c.key && cookie.key.toLowerCase() === c.key.toLowerCase();});diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.tsindex deda02330..65aa76044 100644--- a/test/core/crawlers/cheerio_crawler.test.ts+++ b/test/core/crawlers/cheerio_crawler.test.ts@@ -1180,13 +1180,12 @@ describe('CheerioCrawler', () => {expect(warningSpy).toBeCalledWith(`Found cookies with similar name during cookie merging: 'coo' and 'Coo'`);});- test('mergeCookies() throws a contextual error for malformed cookie fragments', () => {- expect(() => mergeCookies('https://example.com', ['valid=1; brokenfragment'])).toThrow(- 'Could not parse cookie header string: brokenfragment',- );- expect(() => mergeCookies('https://example.com', ['sessionid'])).toThrow(- 'Could not parse cookie header string: sessionid',- );+ test('mergeCookies() skips malformed cookie fragments instead of throwing', () => {+ const warningSpy = vitest.spyOn(BaseCrawleeLogger.prototype, 'warning');+ expect(mergeCookies('https://example.com', ['valid=1; brokenfragment'])).toBe('valid=1');+ expect(mergeCookies('https://example.com', ['a=b', 'c'])).toBe('a=b');+ expect(mergeCookies('https://example.com', ['sessionid'])).toBe('');+ expect(warningSpy).toBeCalled();});test('sendRequest and main request should share the same session cookie jar', async () => {