Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: lock browser-pool version so we do not use latest beta automatically
604459e Martin Adámek committed 5 years ago
fix: headerGeneratorOptions not passed to requestAsBrowser (#1105)
675da8f Ondra Urban committed 5 years ago
fix: hardcode public API URL in key-value store URL getter (#1108)
4e2bd76 Martin Adámek committed 5 years ago
chore: bump pw and puppeteer versions to match chrome 92
5e00a45 Martin Adámek committed 5 years ago
chore: require node 15+ due to http2 issues in older versions (#1099)
e2a5938 Martin Adámek committed 5 years ago
feat: add `forceUrlEncoding` option to `requestAsBrowser` (#1086)
442a7ac Martin Adámek committed 5 years ago
feat: added docs and handling of aborting event into basic crawler (#1052)
9f9e503 Jakub Drobník committed 5 years ago
feat: add `taskTimeoutSecs` to allow control over timeout of `AutoscaledPool` tasks (#1082)
ff6eb3f Martin Adámek committed 5 years ago
test: use local server instead of requests to api.apify.com (#1071)
53604fd Martin Adámek committed 5 years ago
chore: Rename GitHub workflow event for building node images (#1066)
584adbb František Nesveda committed 5 years ago
chore: remove silent flag from jest to see logging for flaky tests
80de678 Ondra Urban committed 5 years ago