Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(release): update internal dependencies [skip ci]
8a15f3b Apify Release Bot committed 2 years ago
refactor: add timeout for adaptive crawler's `enqueueLinks` waitFor call
3780547 Martin Adámek committed 2 years ago
fix(adaptive-crawler): log only once for the committed request handler execution (#2524)
533bd3f Martin Adámek committed 2 years ago
feat: log desired concurrency in the default status message
9f0b796 Martin Adámek committed 2 years ago
fix: increase timeout for retiring inactive browsers (#2523)
195f176 Martin Adámek committed 2 years ago
feat: add `waitForSelector` context helper + `parseWithCheerio` in adaptive crawler (#2522)
6f88e73 Martin Adámek committed 2 years ago
fix: respect implicit router when no `requestHandler` is provided in `AdaptiveCrawler` (#2518)
31083aa Martin Adámek committed 2 years ago
docs: wait before scrolling fragments into viewport (#2517)
0976e18 Jindřich Bär committed 2 years ago