Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
test: use beta docker images and force crawlee from master in apify sdk
dc62fd4 Martin Adámek committed 4 years ago
test: update apify sdk in e2e tests to use the latest dev version (#1524)
4631eac Martin Adámek committed 4 years ago
fix: allow `label` in `enqueueLinksByClickingElements` options (#1525)
18b7c25 Minu Palaniappan committed 4 years ago
test: try to improve error handling around TS compilation
d351b05 Martin Adámek committed 4 years ago
chore: downgrade puppeteer and disable auto updates for it
8d07108 Martin Adámek committed 4 years ago
chore: fix experimental containers firefox test (#1522)
f64f1e2 Szymon Marczak committed 4 years ago
docs: replace Apify SDK with Crawlee in versioned docs (#1517)
1b4d54c Andrey Bykov committed 4 years ago
docs: replace "Apify SDK" with "Crawlee" in core/Request docs (#1515)
c70fdd6 Percival Villalva committed 4 years ago
fix(enqueueLinks): prevent useless request creations when filtering by user patterns (#1510)
cb8fe36 Vlad Frangu committed 4 years ago
fix(enqueueLinks): ensure the enqueue strategy is respected alongside user patterns (#1509)
2b0eeed Vlad Frangu committed 4 years ago