Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Commits on master

feat(utils): add `extractMicrodata` helper (#3233)
Danish Arora committed 2 months ago
a6961b2
feat: instrumentation based opentelemetry collection (#3361)
NathanSavageKaimai committed 2 months ago
3a2c288
fix(basic): report skips from background add batches (#4078)
Michael Bujnovský committed 2 months ago
2436868
feat(core): pass Request to onSkippedRequest (#3992)
Jindřich Bär committed 2 months ago
d26a7f6
feat: drop the `closeCookieModals` helpers (#4073)
Jindřich Bär committed 2 months ago
0369fed
chore: Bump native crawlee storage (#4077)
Jan Buchar committed 2 months ago
0716623
cceca28
ci: cap Build & Test at 30 minutes (#4055)
Martin Adámek committed 2 months ago
a8da146
feat: add `initialConcurrency` crawler option (#4067)
Jindřich Bär committed 2 months ago
45565e2
fix: Deflake tests under high concurrency (#4059)
Jan Buchar committed 3 months ago
73ee2b8
ci: pin third-party actions to commit SHAs (#4051)
Martin Adámek committed 3 months ago
d57a413
chore(deps): security bumps + in-range freshening (#4052)
apify-deps[bot] committed 3 months ago
6d77689
chore(deps): bump pnpm to 11.22.0 (#4050)
Martin Adámek committed 3 months ago
b0e828a