Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix merge conflicts
Ondra Urbane24c86f14d80
diff --cc src/basic_crawler.jsindex fbe2c21ea,f6dfb2132..24f47eac3--- a/src/basic_crawler.js+++ b/src/basic_crawler.js@@@ -66,9 -69,9 +66,9 @@@ const DEFAULT_OPTIONS =* @param {Function} [options.handleRequestFunction]* Function that processes a single `Request` object. It must return a promise.* @param {Function} [options.handleFailedRequestFunction=({ request }) => {-- const details = _.pick(request, 'id', 'url', 'method', 'uniqueKey');- log.error('BasicCrawler: Request failed and reached maximum retries', details);- }]-- log.error('BasicCrawler: Request failed and reached maximum retries', details);]++ * const details = _.pick(request, 'id', 'url', 'method', 'uniqueKey');++ * log.error('BasicCrawler: Request failed and reached maximum retries', details);++ * }]* Function that handles requests that failed more then `option.maxRequestRetries` times.* @param {Number} [options.maxRequestRetries=3]* How many times the request is retried if `handleRequestFunction` failed.