Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
docs: fix blurred logo on homepage in firefox and safari
9cd007b Martin Adámek committed 4 years ago
docs: update getting started and some other guides (#1429)
333faa4 Ondra Urban committed 4 years ago
fix: add missing configuration to CheerioCrawler constructor (#1432)
6e4ab1c Andrey Bykov committed 4 years ago
fix: use `desiredConcurrency: 10` as the default for `CheerioCrawler` (#1428)
0aca5b5 Martin Adámek committed 4 years ago
fix: `enqueueLinks` should respect full URL of the current request for relative link resolution (#1427)
e72d02c Martin Adámek committed 4 years ago
feat: add support for middlewares to the `Router` via `use` method (#1431)
6fee20d Martin Adámek committed 4 years ago
fix: always limit `desiredConcurrency` to the value of `maxConcurrency`
bcb689d Martin Adámek committed 4 years ago
fix: wait for storage to finish before resolving `crawler.run()`
9d62d56 Martin Adámek committed 4 years ago
fix: using explicitly typed router with `CheerioCrawler`
07b7e69 Martin Adámek committed 4 years ago
chore: check the documentation builds on every PR (#1426)
1810103 Martin Adámek committed 4 years ago
feat: allow configuring what status codes will cause session retirement (#1423)
cd3795f Martin Adámek committed 4 years ago
fix: add `UserData` type argument to `CheerioCrawlingContext` and related interfaces (#1424)
a062cb7 Martin Adámek committed 4 years ago