Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
improved puppeteer crawler docs.
Petr Patekfa6663c43070
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.jsindex b861ef135..dd2034066 100644--- a/src/crawlers/puppeteer_crawler.js+++ b/src/crawlers/puppeteer_crawler.js@@ -82,7 +82,8 @@ import { openSessionPool } from '../session_pool/session_pool';* response: Response,* page: Page,* puppeteerPool: PuppeteerPool,- * autoscaledPool: AutoscaledPool+ * autoscaledPool: AutoscaledPool,+ * session: Session,* }* ```*@@ -173,6 +174,13 @@ import { openSessionPool } from '../session_pool/session_pool';* If you're not sure, just keep the default value and the concurrency will scale up automatically.* @param {Object} [options.maxConcurrency=1000]* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.+ * @param {Boolean} [options.useSessionPool=false]+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.+ * It also marks Session as bad after a request timeout.+ * @param {Object} [options.sessionPoolOptions]+ * Custom options passed to the underlying {@link SessionPool} constructor.+ * @param {Boolean} [options.persistCookiesPerSession]+ * Automatically saves cookies to Session. Works only if Session Pool is used.*/class PuppeteerCrawler {constructor(options) {