Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat: don't close page before calling errorHandler (#1548)
1c8cd82 Szymon Marczak committed 4 years ago
fix(http-crawler): do not hang on POST without payload (#1546)
8c87390 Szymon Marczak committed 4 years ago
feat(router): add userdata generic to addHandler (#1547)
19cdf13 Szymon Marczak committed 4 years ago
fix(basic-crawler): handle `request.noRetry` after `errorHandler` (#1542)
2a2040e Matt Stephens committed 4 years ago
fix: add overload for KeyValueStore#getValue with defaultValue (#1541)
e3cb509 Vlad Frangu committed 4 years ago
fix: do not hide stack trace of (retried) Type/Syntax/ReferenceErrors
469b4b5 Martin Adámek committed 4 years ago
feat: add `utils.downloadListOfUrls` to crawlee metapackage
7b33b0a Martin Adámek committed 4 years ago
fix: remove undeclared dependency on core package from puppeteer utils
827ae60 Martin Adámek committed 4 years ago
chore(deps): update actions/checkout action to v3 (#1532)
9e96af8 renovate[bot] committed 4 years ago