Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
9772c1a Apify Release Bot committed 2 years ago
fix: fix crawling context type in `router.addHandler()` (#2355)
d73c202 Martin Adámek committed 2 years ago
chore(release): update internal dependencies [skip ci]
bd430d3 Apify Release Bot committed 2 years ago
chore: remove unused poolPromise property on AutoscaledPool (#2347)
28aba08 Jan Buchar committed 2 years ago
test: do not import puppeteer directly from inside browser pool (#2346)
06ba6ac Martin Adámek committed 2 years ago
feat(ci): snapshot docs automatically on minor/major publish (#2344)
092f51e Jindřich Bär committed 2 years ago
feat(core): add `userAgent` parameter to `RobotsFile.isAllowed()` + `RobotsFile.from()` helper (#2338)
343c159 Martin Adámek committed 2 years ago
fix: `createRequests` works correctly with `exclude` (and nothing else) (#2321)
048db09 Jindřich Bär committed 2 years ago