Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #174 from apifytech/feature/crawler_fixes
e19f62d Marek Trunkát committed 8 years ago
Fixing situation when newly added request is not available in the queue yet causing it to stuck forever
8b3b250 Marek Trunkat committed 8 years ago
Merge pull request #172 from apifytech/feature/pptr_pool_fix
d63dad9 Marek Trunkát committed 8 years ago
Merge pull request #173 from apifytech/feature/puppeteer_crawler_fix
8420650 Marek Trunkát committed 8 years ago
Fixing Puppeteer pool - errors of page.goto() not catched
49085bb Marek Trunkat committed 8 years ago
Fixing Puppeteer pool - errors of page.goto() not catched
e2f6ac0 Marek Trunkat committed 8 years ago
Fixing PuppeteerPool to kill browser when there is one page remaining 'about:blank'
9187b41 Marek Trunkat committed 8 years ago
Merge pull request #171 from apifytech/feature/upgrade-lint
0a21836 Marek Trunkát committed 8 years ago
Upgraded Puppeteer to 1.8.0, upgraded NPM deps, fixed lint errors, other minor fixes
107714b Jan Curn committed 8 years ago
Merge branch 'develop' of github.com:Apifier/apify-runtime-js into develop
ff2af8b Marek Trunkat committed 8 years ago
Merge branch 'develop' of github.com:Apifier/apify-runtime-js into develop
a01ead5 Marek Trunkat committed 8 years ago
Merge pull request #167 from apifytech/feature/better-readme
8e0f0da Marek Trunkát committed 8 years ago