Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Move constants ENV_VARS, DEFAULT_PROXY_HOSTNAME, DEFAULT_PROXY_PORT t… (#73)
1a39cb3 Jakub Drobník committed 8 years ago
Moving some constants to apify-shared and changing proxy configuration (#71)
0520179 Marek Trunkát committed 8 years ago
Changing logging to plaintext and updating some messages (#67)
55ca0ce Marek Trunkát committed 8 years ago
Renamed abortInstanceAfterRequestCount to retireInstanceAfterRequestC… (#64)
d1b8eda Jan Čurn committed 8 years ago
Fixing validation of RequestQueue parameter to also allow RequestQueueLocal (#60)
81ded9c Marek Trunkát committed 8 years ago
Logging failed requests in basic crawler - #fixes #46 (#58)
f605715 Marek Trunkát committed 8 years ago
Allowing to override is finished function of BasicCrawler and PuppeteerCrawler (#55)
fb36c2a Marek Trunkát committed 8 years ago
Fixing problem with BasicCrawler getting stucked and adding unit tests
62ff1d6 Marek Trunkat committed 8 years ago