Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: use config instance in `sdk.openSessionPool()` (#1154)
664bdc9 Martin Adámek committed 5 years ago
fix: add ts-ignore comments to imports of optional peer dependencies (#1152)
bb9ad87 Martin Adámek committed 5 years ago
chore: mark `optionsShape` property as internal to fix typings (#1150)
ce0d63d Martin Adámek committed 5 years ago
fix: casting of int/bool environment variables (e.g. `APIFY_LOCAL_STORAGE_ENABLE_WAL_MODE`) (#1146)
090bbbe Martin Adámek committed 5 years ago
fix: deprecation messages in `CheerioCrawler` and `ProxyConfiguration` (#1129)
7b7a971 Ondra Urban committed 5 years ago
chore: Use `DISABLE_OUTDATED_WARNING` from `@apify/consts` instead of defining it separately (#1127)
81fe80f František Nesveda committed 5 years ago
fix: limit `handleRequestTimeoutMillis` to max valid value (#1116)
5948958 Szymon Marczak committed 5 years ago