Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(core): conversion between tough cookies and browser pool cookies (#2443)
74f73ab Gustavo Silva committed 2 years ago
fix(core): use createSessionFunction when loading Session from persisted state (#2444)
3c56b4c Gustavo Silva committed 2 years ago
fix: `EnqueueStrategy.All` erroring with links using unsupported protocols (#2389)
8db3908 Stefan Sundin committed 2 years ago
fix: do not drop statistics on migration/resurrection/resume (#2462)
8ce7dd4 Jan Buchar committed 2 years ago
feat: make `RequestQueue` v2 the default queue, see more on [Apify blog](https://blog.apify.com/new-apify-request-queue/) (#2390)
41ae8ab Vlad Frangu committed 2 years ago
perf: optimize adding large amount of requests via `crawler.addRequests()` (#2456)
6da86a8 Martin Adámek committed 2 years ago
chore(deps): update dependency linkedom to ^0.18.0 (#2457)
420fbc7 renovate[bot] committed 2 years ago
fix(core): fire local `SystemInfo` events every second (#2454)
1fa9a66 Martin Adámek committed 2 years ago
fix: sitemap `content-type` check breaks on `content-type` parameters (#2442)
db7d372 Jindřich Bär committed 2 years ago
docs: fix two typos (array or requests -> array of requests, no much -> not much) (#2451)
1de6d0f Jiří Spilka committed 2 years ago
docs: Should be "Same Domain" not "Same Subdomain" (#2445)
63b9949 Connor Adams committed 2 years ago