Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Commented out launchChrome() function as well as lighthouse and chrome-remote-interface dependencies, updated README.md
978926c Jan Curn committed 9 years ago
Development of .travis.yml, commented out unit tests for launchChrome()
31404d7 Jan Curn committed 9 years ago
Updated utils.newClient() to only use the new apify-client 'baseUrl' option, updated unit test
e1ef481 Jan Curn committed 9 years ago
Ensure package is published by calling publish.sh, not by 'npm publish'
7f44e25 Jan Curn committed 9 years ago
Updated unit tests to support new apify-client's baseUrl setting, bumped package.json version
b0fe81c Jan Curn committed 9 years ago