Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Add mime package to support all known content-type extensions
47e69ff Ondra Urban committed 8 years ago
Merge pull request #138 from apifytech/feature/split-dataset-payloads
cc32d10 Ondra Urban committed 8 years ago
Bump version of apify-shared to get MAX_PAYLOAD_SIZE_BYTES constant
9d0773b Ondra Urban committed 8 years ago
Add max payload size validation and chunking to pushData()
b234d7d Ondra Urban committed 8 years ago
Merge pull request #133 from apifytech/bug/invalid-url-regex
34628bf Marek Trunkát committed 8 years ago
Merge pull request #136 from apifytech/feature/add-live-view-docs
abce5a0 Marek Trunkát committed 8 years ago