Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: improve error message when no default route provided
04c3b6a Martin Adámek committed 4 years ago
fix: add `headless` option to `BrowserCrawlerOptions` (#1412)
7896839 Martin Adámek committed 4 years ago
fix: ensure default route is not ignored in `CheerioCrawler` (#1411)
6475628 Martin Adámek committed 4 years ago
fix: rename interface `FailedRequestHandler` to `ErrorHandler` (#1410)
c5b74c6 Martin Adámek committed 4 years ago
fix: run browser crawlers in headless mode by default (#1409)
b207c12 Martin Adámek committed 4 years ago
fix: calling enqueueLinks in browser crawler on page without any links
385ca27 Martin Adámek committed 4 years ago
chore(deps): bump terser from 5.13.1 to 5.14.2 in /website (#1408)
00c2ba3 dependabot[bot] committed 4 years ago
Revert "chore(deps): update dependency marked to 4.0.10 [security] (#1407)"
c22da4f Martin Adámek committed 4 years ago
chore(deps): update dependency marked to 4.0.10 [security] (#1407)
2185f26 renovate[bot] committed 4 years ago
fix: rename default storage folder to just `storage` (#1403)
c253985 Martin Adámek committed 4 years ago
fix: remove `JSONData` generic type arg from `CheerioCrawler` (#1402)
f8d486e Martin Adámek committed 4 years ago
docs: update introduction guides, update screenshots (#1394)
57c9cbe Andrey Bykov committed 4 years ago
docs: do not document deprecated options like `handlePageFunction`
ec1e0d2 Martin Adámek committed 4 years ago
chore(deps): update dependency @docusaurus/preset-classic to v2.0.0-rc.1
5681810 renovate[bot] committed 4 years ago