Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: only provide Cookie header if cookies are present (#1218)

Vlad Frangucfdcbcb87fc2
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 984615417..2c0d06e9c 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -705,11 +705,13 @@ class CheerioCrawler extends BasicCrawler {
_applySessionCookie({ request, session }, requestAsBrowserOptions) {
const userCookie = request.headers.Cookie ?? request.headers.cookie;
const sessionCookie = session.getCookieString(request.url);
+ const mergedCookies = mergeCookies(request.url, [sessionCookie, userCookie]);
// merge cookies from all possible sources
- requestAsBrowserOptions.headers = {
- Cookie: mergeCookies(request.url, [sessionCookie, userCookie]),
- };
+ if (mergedCookies) {
+ requestAsBrowserOptions.headers ??= {};
+ requestAsBrowserOptions.headers.Cookie = mergedCookies;
+ }
}
/**