Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Allow use of already required modules as 'puppeteerModule' option.
ea081e4 Ondra Urban committed 7 years ago
Merge pull request #364 from apifytech/fix/prepare-request
4ec9646 Ondra Urban committed 7 years ago
Merge pull request #365 from apifytech/feature/remove-load-scheduler
f5cf691 Ondra Urban committed 7 years ago
Merge pull request #368 from apifytech/feature/live-view-ui
9785c47 Ondra Urban committed 7 years ago
Make the usage smoother and improve test performance by adding static html.
4e8c2dd Petr Patek committed 7 years ago
fixed travis configuration for chrome. The chrome beta directory is deprecated in travis so it downloads chrome version 64.
9d71aa6 Petr Patek committed 7 years ago
Removed --enable-resource-load-scheduler=false Chrome command line flag
70ddee4 Jan Curn committed 7 years ago
Change back to first page, because it needs to be in focus
357423f Ondra Urban committed 7 years ago