Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(core): log a warning on malformed cookie fragments in `mergeCookies` (#3850)

Jindřich Bär46313998b820
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index adc0391c0..a84fdcad1 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -739,6 +739,8 @@ const response = await sendRequest({ url: '...' }, { cookieJar: jar });
The protected `HttpCrawler._applyCookies` method is removed. If you were overriding it in a subclass, move your logic to a `preNavigationHook` that sets cookies on `request.headers.Cookie` or on the `session` cookie jar directly.
+`mergeCookies` now skips malformed cookie fragments with a warning instead of throwing.
+
### `Session.getCookies`, `setCookies` and `setCookiesFromResponse` are removed
The public cookie helper methods on `Session` — `getCookies(url)`, `setCookies(cookies, url)`, and `setCookiesFromResponse(response)` — have been removed as part of centralizing cookie assembly in `BaseHttpClient`. Work with the session's `cookieJar` directly, or use the new `Session.getCookieString(url)` to read the assembled `Cookie` header value.
diff --git a/packages/core/src/cookie_utils.ts b/packages/core/src/cookie_utils.ts
index 53aeb9130..4213f9f19 100644
--- a/packages/core/src/cookie_utils.ts
+++ b/packages/core/src/cookie_utils.ts
@@ -120,7 +120,10 @@ export function mergeCookies(url: string, sourceCookies: string[]): string {
if (!cookieString) continue;
const cookie = Cookie.parse(cookieString);
- if (!cookie) throw new CookieParseError(cookieString);
+ if (!cookie) {
+ serviceLocator.getLogger().warning(`Skipping malformed cookie fragment: '${cookieString}'`);
+ continue;
+ }
const similarKeyCookie = jar.getCookiesSync(url).find((c) => {
return cookie.key !== c.key && cookie.key.toLowerCase() === c.key.toLowerCase();
});
diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.ts
index deda02330..65aa76044 100644
--- a/test/core/crawlers/cheerio_crawler.test.ts
+++ b/test/core/crawlers/cheerio_crawler.test.ts
@@ -1180,13 +1180,12 @@ describe('CheerioCrawler', () => {
expect(warningSpy).toBeCalledWith(`Found cookies with similar name during cookie merging: 'coo' and 'Coo'`);
});
- test('mergeCookies() throws a contextual error for malformed cookie fragments', () => {
- expect(() => mergeCookies('https://example.com', ['valid=1; brokenfragment'])).toThrow(
- 'Could not parse cookie header string: brokenfragment',
- );
- expect(() => mergeCookies('https://example.com', ['sessionid'])).toThrow(
- 'Could not parse cookie header string: sessionid',
- );
+ test('mergeCookies() skips malformed cookie fragments instead of throwing', () => {
+ const warningSpy = vitest.spyOn(BaseCrawleeLogger.prototype, 'warning');
+ expect(mergeCookies('https://example.com', ['valid=1; brokenfragment'])).toBe('valid=1');
+ expect(mergeCookies('https://example.com', ['a=b', 'c'])).toBe('a=b');
+ expect(mergeCookies('https://example.com', ['sessionid'])).toBe('');
+ expect(warningSpy).toBeCalled();
});
test('sendRequest and main request should share the same session cookie jar', async () => {