Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat(router): allow inline router definition (#1877)
2d241c9 Pavel 'Strajk' Dolecek committed 3 years ago
docs: fix code examples for Adding more URLs - Finding new links and Filtering links to same domain (#1884)
f4b0c62 Eirik Mo committed 3 years ago
fix: respect config object when creating `SessionPool` (#1881)
db069df Martin Adámek committed 3 years ago
feat: allow running single crawler instance multiple times (#1844)
9e6eb1e Martin Adámek committed 3 years ago
build: do not fail publishing latest when there is nothing to commit
beda03e Martin Adámek committed 3 years ago
fix(Storage): queue up opening storages to prevent issues in concurrent calls (#1865)
044c740 Vlad Frangu committed 3 years ago