Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update styfle/cancel-workflow-action action to v0.10.0 (#1385)
b900ade renovate[bot] committed 4 years ago
chore: use apify.com as the default start url in project templates
5e8428d Martin Adámek committed 4 years ago
chore(deps): update styfle/cancel-workflow-action action to v0.10.0 (#1379)
1a7a93b renovate[bot] committed 4 years ago
fix: update utils.enqueueLinks to ignore invalid URLs when using cheerio (#1369)
953fb66 Ashley Evans committed 4 years ago
Revert "chore(deps): update dependency puppeteer to v14" (#1356)
55e0cce Martin Adámek committed 4 years ago
fix: update utils.purgeLocalStorage to mimic 'purge' param of 'apify run' (#1355)
b9b77cc Andrey Bykov committed 4 years ago
fix: use default user agent for playwright with chrome (#1350)
d2d5ac7 Martin Adámek committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v4.5.0 (#1349)
9b7d29c renovate[bot] committed 4 years ago
fix: random puppeteer crawler (running in headful mode) failure (#1348)
e2d9233 Andrey Bykov committed 4 years ago
fix: ensure failed req count is correct when using `RequestList` (#1347)
8090509 Ondra Urban committed 4 years ago
feat: add `keyValueStore` option to `Statistics` class (#1345)
898cc43 Martin Adámek committed 4 years ago