Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Updated comments, added new TODOs, added new (failing) unit tests
60f5197 Jan Curn committed 9 years ago
Renamed environment variables to match the server, bumped package.json version
0e22ddc Jan Curn committed 9 years ago
Improved unit tests for Apifier.main(), better handling of GET requests
0383d1f Jan Curn committed 9 years ago
Consolidated body parsing middlewares, added unit tests, updated docs and bumped package version; removed yarn.lock from git
8922e84 Jan Curn committed 9 years ago
Replaced .npmignore with "files" directive in package.json, bumped package.json version
b62a9d8 Jan Curn committed 9 years ago
Added .npmignore to .npmignore (inception style), bumped package version
1be5538 Jan Curn committed 9 years ago
Fixed a bug - old reference to Actor.heyIAmReady(), added a test case for that to unit tests, bumped package.json version
983d127 Jan Curn committed 9 years ago
Merge branch 'master' of https://github.com/Apifier/apifier-sdk-js
85ec4e3 Jan Curn committed 9 years ago
Added Apifier.setPromisesDependency() and Apifier.getPromisesDependency()
29c38da Jan Curn committed 9 years ago