Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
added tests for cookies and moved update cookies function from CheerioCrawler to utils.
4a306e1 Petr Patek committed 7 years ago
CheerioCrawler SessionPool implementation and Basic CookieJar per session logic.
69d2616 Petr Patek committed 7 years ago
Merge pull request #513 from apifytech/feature/case-insensitive-merge
e5a1869 Ondra Urban committed 7 years ago
Merge pull request #512 from apifytech/fix/infinite-scroll
8f34d32 Ondra Urban committed 7 years ago
Merge pull request #511 from apifytech/fix/cheerio-encoding
99d7c29 Ondra Urban committed 7 years ago
Fix CheerioCrawler when other than Node supported encoding is used
1dea5c8 Ondra Urban committed 7 years ago
CheerioCrawler: Added doc for option json in handlePageFunction
1e6f2e8 drobnikj committed 7 years ago
Merge pull request #477 from apifytech/feature/session-pool-basic-crawler
5028b18 Ondra Urban committed 7 years ago
Merge branch 'master' into feature/session-pool-basic-crawler
6b86f60 Petr Patek committed 7 years ago
Merge pull request #494 from apifytech/feature/add_yelp_youtube_social
96f97b5 Ondra Urban committed 7 years ago
Merge pull request #495 from apifytech/feature/save-snapshot
75dc619 Ondra Urban committed 7 years ago
Merge pull request #496 from apifytech/feature/petr-proxy-country-url
8ef15ff Ondra Urban committed 7 years ago
Merge branch 'master' into feature/add_yelp_youtube_social
13a5bf8 Ondra Urban committed 7 years ago