Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
CheerioCrawler - reference the session pool options to underlying basic crawler (#749)
petrpatek3e13f3a636a7
diff --git a/src/constants.js b/src/constants.jsindex 375fec6ae..b178b743a 100644--- a/src/constants.js+++ b/src/constants.js@@ -4,7 +4,8 @@ import { ACTOR_EVENT_NAMES } from 'apify-shared/consts';* The default user agent used by `Apify.launchPuppeteer`.* Last updated on 2020-05-22.*/-export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36';+// eslint-disable-next-line max-len+export const DEFAULT_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36';/*** Exit codes for the actor process.diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex d82887fe5..c0b5ad90b 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -389,8 +389,8 @@ class CheerioCrawler {autoscaledPoolOptions,// Session pool options- sessionPoolOptions,- useSessionPool,+ sessionPoolOptions: this.sessionPoolOptions,+ useSessionPool: this.useSessionPool,// loglog: this.log,