Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: make the status message interval 10s and add initial message (#1920)
5ac530a Martin Adámek committed 3 years ago
fix: set status message every 5 seconds and log it via debug level (#1918)
32aede6 Martin Adámek committed 3 years ago
feat: add support for `requestsFromUrl` to `RequestQueue` (#1917)
7f2557c Martin Adámek committed 3 years ago
docs: minor syntax errors in blockRequests docs example (#1914)
1f1ed2a Brian Hann committed 3 years ago
chore: disable puppeteer deprecation warning about old `headless` mode
15c7f67 Martin Adámek committed 3 years ago
chore(deps): update dependency htmlparser2 to v9 (#1908)
2170c27 renovate[bot] committed 3 years ago
feat: support alternate storage clients when opening storages (#1901)
661e550 Vlad Frangu committed 3 years ago
feat(HttpCrawler): add `parseWithCheerio` helper to `HttpCrawler` (#1906)
ff5f76f Pavel 'Strajk' Dolecek committed 3 years ago
fix(MemoryStorage): cache requests in `RequestQueue` (#1899)
063dcd1 Vlad Frangu committed 3 years ago