Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(core): solve possible dead locks in `RequestQueueV2` (#2376)
ffba095 Vlad Frangu committed 2 years ago
chore: fix internal attribute naming in SystemStatus (#2373)
8ba0f30 Vlada Dusek committed 2 years ago
feat: implement global storage access checking and use it to prevent unwanted side effects in adaptive crawler (#2371)
fb3b7da Jan Buchar committed 2 years ago
fix: use 0 (number) instead of false as default for sessionRotationCount (#2372)
667a3e7 Jan Buchar committed 2 years ago
chore(deps): update dependency @giscus/react to v3 (#2363)
af6e8c1 renovate[bot] committed 2 years ago