Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: suppress info message for undefined maxRequestsPerCrawl (#3237)
f3d9a79 Rafael Laidlaw committed 1 year ago
chore(release): update internal dependencies [skip ci]
615c8f9 Apify Release Bot committed 1 year ago
feat: support custom `userAgent` with `respectRobotsTxtFile` (#3226)
354252d Jindřich Bär committed 1 year ago
fix: use shared enqueue links wrapper in `AdaptivePlaywrightCrawler` (#3188)
9569d19 Lukáš Průša committed 1 year ago
chore(deps): update dependency camoufox-js to ^0.8.0 (#3224)
29acbad renovate[bot] committed 1 year ago
fix(cli): support creating projects with names that start with a number (#3219)
3f37845 Danish Arora committed 1 year ago
chore(release): update internal dependencies [skip ci]
c7d788f Apify Release Bot committed 1 year ago