Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge branch 'master' of github.com:Apifier/apify-runtime-js into develop
647550b Marek Trunkat committed 8 years ago
launchPuppeteer() uses more intelligent default User-Agent (#52)
119df61 Jan Čurn committed 8 years ago
Basic Crawler: add error to handleFailedRequestFunction (#50)
0d96150 Jakub Drobník committed 8 years ago
Added useChrome option to launchPuppeteer() + unit test, renamed unit… (#45)
b71ab73 Jan Čurn committed 8 years ago
Adding finishWhenEmpty and maybeRunIntervalMillis opts to autoscaled pool (#41)
0f91e26 Marek Trunkát committed 8 years ago
Don't displaying errors when instance is killed for reason (#42)
96711a2 Marek Trunkát committed 8 years ago