Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: make `format:check` actually check the formatting (#3858)
8f95bb2 Martin Adámek committed 4 months ago
feat: make `cheerio` and `proxy-chain` imports dynamic (#3836)
fcca6fe Jindřich Bär committed 4 months ago
feat: unify extended unique key format with Python, add `alwaysEnqueue` (#3840)
8eab134 Jindřich Bär committed 4 months ago
ci: exclude `@crawlee/fs-storage-native` from canary/pin version rewrite (#3844)
6c50d37 Martin Adámek committed 4 months ago
refactor!: Rename `StorageClient` to `StorageBackend` (#3825)
d7e44dc Jan Buchar committed 4 months ago
fix(memory-storage): store JSON snapshots in `DatasetClient.pushData()` (#3831)
d0c5803 PilgrimStack committed 4 months ago
refactor!: Move status message handling from StorageClient to the event system (#3818)
d73291e Jan Buchar committed 4 months ago
refactor!: Rewrite the FilesystemStorageClient to use apify/crawlee-storage (#3762)
2944b32 Jan Buchar committed 4 months ago
fix: Rectify inconsistencies in the StorageClient interface family (#3800)
a57768b Jan Buchar committed 4 months ago
fix: Fix repeated Storage.open() using the legacy idOrName parameter (#3808)
43effa1 Jan Buchar committed 4 months ago
fix(e2e): repair E2E test infrastructure for pnpm + crawlee v4 (#3805)
08ac4fe Jan Buchar committed 4 months ago
fix: Prevent browser-based crawling mode in AdaptivePlaywrightCrawler from swallowing logs (#3803)
50293fe Jan Buchar committed 4 months ago
fix: `RequestManagerTandem` returns correct total count (#3804)
6c2fec0 Jindřich Bär committed 4 months ago
refactor!: Dissolve @crawlee/memory-storage into @crawlee/core (#3802)
5e2c92d Jan Buchar committed 4 months ago
fix: disable storage access on `requestHandler` timeouts (#3788)
4f0c4c4 Jindřich Bär committed 4 months ago
fix: read content encoding hints from `<meta>` tags (#3780)
1dc8812 Jindřich Bär committed 4 months ago
feat: rename KVS objects from `SDK_` to `CRAWLEE_` (#3783)
8af187d Jindřich Bär committed 4 months ago
refactor!: Centralize KVS value semantics in @crawlee/core (#3774)
c067c50 Jan Buchar committed 4 months ago
docs: reflect the recent `SessionPool` changes in a guide (#3724)
60e9e36 Jindřich Bär committed 4 months ago
refactor!: Split MemoryStorage into FileSystemStorageClient and MemoryStorageClient (#3740)
2f0a60d Jan Buchar committed 5 months ago
refactor!: Align `RequestQueueClient` interface with its Python counterpart (#3729)
32c0e74 Jan Buchar committed 5 months ago