Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Allow AutoscaledPool to run at minConcurrency when overloaded
485b8c3 Ondra Urban committed 8 years ago
Upgraded @apify/ps-tree depedency (fixes "Error: spawn ps ENFILE"), upgraded other NPM packages
9b3279d Jan Curn committed 8 years ago
Merge branch 'develop' of https://github.com/apifytech/apify-js into develop
ac13875 Jan Curn committed 8 years ago
Updated README, added CONTRIBUTING.md, renamed LICENSE to LICENSE.md
a264fbc Jan Curn committed 8 years ago