Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore: add note regarding puppeteer error to changelog (#1331)
c5b2934 Andrey Bykov committed 4 years ago
chore: use `handleRequestTimeoutMillis * 2` when computing internal timeout value
4f54355 Martin Adámek committed 4 years ago
feat: allow passing a stream to `KeyValueStore.setRecord` (#1325)
31ebd5c Gabin Desserprit committed 4 years ago
fix: reset `RequestQueue` state after 5 minutes of inactivity (#1324)
fc6e4bf Martin Adámek committed 4 years ago
feat: requestList accepts proxyConfiguration for `requestsFromUrls` (#1317)
c120837 Jindřich Bär committed 4 years ago
chore(deps): update peter-evans/repository-dispatch action to v2 (#1316)
daf446d renovate[bot] committed 4 years ago
chore: backport https://github.com/apify/apify-ts/pull/57 (#1312)
121ec85 Vlad Frangu committed 4 years ago
chore(deps): update amannn/action-semantic-pull-request action to v4 (#1289)
75eef82 renovate[bot] committed 4 years ago