Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(jsdom): add timeout to the window.load wait when `runScripts` are enabled
806de31 Martin Adámek committed 3 years ago
refactor: do not add multiple jsdomError handlers to the virtual console
222ad7a Martin Adámek committed 3 years ago
fix(jsdom): use no-op `enqueueLinks` in http crawlers when parsing fails
fd35270 Martin Adámek committed 3 years ago
refactor: don't fail with warning when finding links via JSDOM fails
14d2756 Martin Adámek committed 3 years ago
refactor: handle `jsdomError` event in JSDOM virtual console
dc60f0c Martin Adámek committed 3 years ago
fix(jsdom): delay closing of the window and add some polyfills
2e81618 Martin Adámek committed 3 years ago
fix(templates): added missing '@types/node' peer dependency (#1860)
d37a7e2 Alex committed 3 years ago
fix(MemoryStorage): handling of readable streams for key-value stores when setting records (#1852)
a5ee37d Vlad Frangu committed 3 years ago
feat: add `parseWithCheerio` context helper to cheerio crawler
b336a73 Martin Adámek committed 3 years ago
fix: start status message logger after the crawl actually starts
5d1df7a Martin Adámek committed 3 years ago
chore: mark `_applyCookies` as protected to resolve issues with type intersections
9c670a6 Martin Adámek committed 3 years ago
test: fix storage pagination e2e tests after website redesign
29929df Martin Adámek committed 3 years ago