Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Adding timeout that ensures that chrome will be really killed in PuppeteerPool
5497adb Marek Trunkat committed 8 years ago
Replacing Math.random() with cryptoRandomObjectId() in proxy sessionId generation
e166b85 Marek Trunkat committed 8 years ago
Merge branch 'develop' of https://github.com/apifytech/apify-js into develop
e25d402 Jan Curn committed 8 years ago
Adding this.minFreeMemoryRatio configuration to AutoscaledPool
954b28c Marek Trunkat committed 8 years ago
Adding this.minFreeMemoryRatio configuration to AutoscaledPool
be02359 Marek Trunkat committed 8 years ago
Upgraded puppeteer optional dependency, bumped package.json version, updated CHANGELOG.md
9abe0cf Jan Curn committed 8 years ago
Merge pull request #22 from Apifier/feature/rename-sequential-store-to-dataset
4a24384 Marek Trunkát committed 8 years ago
Merge remote-tracking branch 'origin/develop' into feature/rename-sequential-store-to-dataset
dbec659 Jaroslav Hejlek committed 8 years ago
Bugfix of Apify.call() - add delay if getRun() returns null, added unit test
82a1699 Jan Curn committed 8 years ago