Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Fixed parsing for bad content-type header in CheerionCrawler #498
d24d641 drobnikj committed 7 years ago
fix retryHistogram for RequestList when URL is the same
e33cd25 metalwarrior665 committed 7 years ago
Add --detectOpenHandles because tests don't exit on Travis
8fbea91 Ondra Urban committed 7 years ago
Merge pull request #487 from apifytech/feature/call-task-458
75d0050 Ondra Urban committed 7 years ago
Merge pull request #491 from apifytech/feature/adhoc-webhook-472
5c432a5 Ondra Urban committed 7 years ago
Merge pull request #492 from apifytech/fix/rq-local-race-condition
66aff32 Ondra Urban committed 7 years ago
Merge pull request #493 from apifytech/feature/request_list_persist_downloaded_sources
f0f94d2 Ondra Urban committed 7 years ago