Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
Merge pull request #310 from apifytech/feature/queue-get-info
c769c40 Ondra Urban committed 7 years ago
Deprecated AutoscaledPools concurrency functions and replaced them with getters/setters
03fd103 Jan Curn committed 7 years ago
(Dataset|RequestQueue).getInfo() no longer accept an options argument, fixed metadate updates, updated comments and tests
df61113 Jan Curn committed 7 years ago
Fixed LocalDataset.getInfo() function to return cleanItemsCount, fixed docs and unit tests
edd6773 Jan Curn committed 7 years ago
Added RequestQueue.getInfo() function, bugfixed LocalRequestQueue.getRequest() to return null if not found, improved docs and tests
7582afb Jan Curn committed 7 years ago
Improved Apify.main() to provide nicer stack traces on errors, fixed unit tests
e208977 Jan Curn committed 7 years ago
Merge pull request #305 from apifytech/feature/upgrade-puppeteer
4f25e0f Ondra Urban committed 7 years ago
Added AutoscaledPool.setDesiredConcurrency() function, improved docs
54ecf8e Jan Curn committed 7 years ago