Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #436 from apifytech/chore/refactor-delete-to-drop
3b2b80d Ondra Urban committed 7 years ago
Merge pull request #431 from apifytech/feature/check-chrome-user-agent
fc5a1cf Ondra Urban committed 7 years ago
Merge branch 'master' into feature/check-chrome-user-agent
40cbefe Ondra Urban committed 7 years ago
Add warning log to enqueueByClicking when the element is detached
719747f Ondra Urban committed 7 years ago
Refactored cheerioCrawler to use requestAsBrowser. Lowered eslint to v5
f38a041 Petr Patek committed 7 years ago
Merge pull request #434 from apifytech/chore/eslint-config
2c820ea Ondra Urban committed 7 years ago
Merge pull request #432 from apifytech/feature/puppeteer-crawler-timeout
854c2f6 Ondra Urban committed 7 years ago
Merge branch 'master' into feature/puppeteer-crawler-timeout
56e3a32 Ondra Urban committed 7 years ago
Merge pull request #429 from apifytech/feature/version_warning
3029b0c Ondra Urban committed 7 years ago
Merge pull request #428 from apifytech/fix/update-user-agent
d89a061 Ondra Urban committed 7 years ago
Merge pull request #421 from apifytech/feature/request_as_browser_update
b0bbe61 Ondra Urban committed 7 years ago