Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
test: adapt E2E suite to v4 and fix regressions it caught (#4012)
ea9da4c Martin Adámek committed 3 months ago
refactor: Use RecoverableState to implement Statistics persistence (#4014)
f08e8d6 Jan Buchar committed 3 months ago
feat: Extend RecoverableState API to accommodate future adopters (#4006)
0d88042 Jan Buchar committed 3 months ago
chore: support RC releases in the publish workflow (#4009)
bb2f267 Martin Adámek committed 3 months ago
chore: wire the `ignoreTlsErrors` crawler option (formerly `ignoreSslErrors`) through to the HTTP client (#4011)
d48cd5b Martin Adámek committed 3 months ago
feat: implement per-domain request throttling (ThrottlingRequestManager) (#3741)
7bca635 Harry committed 3 months ago
feat(utils): split exports into public and /internal entry points (#3991)
6a94b86 Jindřich Bär committed 3 months ago
fix(browser-pool): make PlaywrightBrowser.browserType() return the browser type (#3981)
d95cf20 Martin Adámek committed 3 months ago
fix: declare own `CookieJar` types in `@crawlee/types` (#3978)
dd7748d Jindřich Bär committed 3 months ago
fix: don't ship `idcac-playwright` as a hard dependency (#3977)
c596d23 Jindřich Bär committed 3 months ago
refactor!: use native private class fields and remove underscore prefixes (#3980)
4d14072 Martin Adámek committed 3 months ago
docs: Document LoadSignal customization for ConcurrencySystem (#3976)
80effc1 Jan Buchar committed 3 months ago
feat: add user data generics to browser navigation hooks (#3970)
28e24b4 Martin Adámek committed 3 months ago
refactor!: Accept a pre-configured Statistics instance in crawlers instead of an option bag (#3966)
d915e53 Jan Buchar committed 3 months ago
test: fix flaky gotoOptions assertion in browser crawler test (#3973)
6a1ea15 Martin Adámek committed 3 months ago
docs: Update SessionPool section in upgrading guide (#3974)
a99f894 Jan Buchar committed 3 months ago
docs: document removal of gotoOptions argument from browser navigation hooks (#3972)
f2ef899 Martin Adámek committed 3 months ago
perf: lazy-load `cheerio` in `playwright`/`puppeteer` `parseWithCheerio` (#3968)
cdab1b1 Jindřich Bär committed 3 months ago
refactor: Remove unnecessary retries when enqueueing new requests (#3962)
f2f4397 Jan Buchar committed 3 months ago