Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Changed apifier.com to apify.com, bumped package.json version
a858625 Jan Curn committed 9 years ago
Merge pull request #8 from Apifier/feature/kv_store_charset
94f625a Marek Trunkát committed 9 years ago
Adding charset=utf-8 when storing the record to key value store
a316b10 Marek Trunkat committed 9 years ago
Bugfix in Apify.setValue() - APIFY_DEFAULT_KEY_VALUE_STORE_ID env var is not always required, Apify.getValue() also supports 'application/json; charset=utf-8' content type, improved unit tests
2ee306d Jan Curn committed 9 years ago
Fixes in handling of APIFY_DEV_KEY_VALUE_STORE_DIR env var - check directory exists, getValue() returns null if file does not exist
e910733 Jan Curn committed 9 years ago
Change path.join() to path.resolve() to have full paths, bumped package.json version
faa7db9 Jan Curn committed 9 years ago