Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #303 from apifytech/feature/queue_kvstore_fixexz
3d77672 Ondra Urban committed 7 years ago
Merge pull request #292 from apifytech/feature/request-loadedUrl
a223d19 Ondra Urban committed 7 years ago
Merge pull request #296 from novotnyj/fix/kv-store-forEachKey2
f845335 Ondra Urban committed 7 years ago
Merge pull request #295 from apifytech/feature/petr-memory-leak-log
6c5b17d Ondra Urban committed 7 years ago
Fixing KV store bugs causing unhandled promise rejection
a88f4aa Marek Trunkat committed 7 years ago
Fixing request queue - repeating of queries for consistency
901d6fa Marek Trunkat committed 7 years ago