Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: `SitemapRequestList.teardown()` doesn't break `persistState` calls (#2673)
fb2c5cd Jindřich Bär committed 2 years ago
docs: changes regarding the updated structure of py templates (#2667)
238d929 Vlada Dusek committed 2 years ago
chore(release): update internal dependencies [skip ci]
02847d4 Apify Release Bot committed 2 years ago
ci: fix turbo and yarn incompatibility with node 16 (#2657)
5feca1b Vlad Frangu committed 2 years ago
fix(RequestQueueV2): reset recently handled cache too if the queue is pending for too long (#2656)
51a69bc Vlad Frangu committed 2 years ago
fix: improve `FACEBOOK_REGEX` to match older style page URLs (#2650)
a005e69 Joe Leonard committed 2 years ago
docs: use green for code highlighting + fix code line numbers on long lines
811d194 Martin Adámek committed 2 years ago