Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(playwright): track and drain in-flight rendering detections on teardown (#4100)
8abbb5d Yash Hirgude committed 2 months ago
refactor(core)!: persist RequestList and SessionPool state through RecoverableState (#4068)
b70693f Vlad Frangu committed 2 months ago
chore: Update queue purging documentation; Remove worthless e2e test (#4102)
8939514 Jan Buchar committed 2 months ago
fix(core): stop scanning handled requests in the in-memory request queue (#4083)
fde8d84 Vlad Frangu committed 2 months ago
fix!: Don't empty the request queue between run() calls (#4056)
5656fed Jan Buchar committed 2 months ago
chore: pin optionalDependencies in canary and release version pinning (#4098)
ef0b25a Martin Adámek committed 2 months ago
fix(basic): isolate crawler.run() from the caller's storage transaction (#4093)
c3bd464 Jindřich Bär committed 2 months ago
chore(deps): dependency sweep — security bump + in-range freshening (#4095)
e9286be apify-deps[bot] committed 2 months ago
feat: instrumentation based opentelemetry collection (#3361)
3a2c288 NathanSavageKaimai committed 2 months ago
fix(playwright): wire browser options through AdaptivePlaywrightCrawler (#4088)
6cc2420 Jindřich Bär committed 2 months ago
fix(basic): report skips from background add batches (#4078)
2436868 Michael Bujnovský committed 2 months ago
docs: add a guide on dismissing cookie modals in Crawlee crawlers (#4091)
d405370 Jindřich Bär committed 2 months ago
docs: generate markdown versions for all documentation versions (#4090)
25d01d5 Jindřich Bär committed 2 months ago
fix(core): honour domain backoff for requests the wrapped manager holds (#4040)
dbe57fb Mikhail Koviazin committed 2 months ago
refactor: keep same-domain filtering after an off-domain redirect (#4076)
bd4771e Martin Adámek committed 2 months ago
chore: move the api surface extractor to its own repo (#4045)
cceca28 Jindřich Bär committed 2 months ago
refactor!: Add checkReadiness() and recordPacingSignal() to IRequestManager to fix leaky abstraction of ThrottlingRequestManager (#4061)
6d36638 Jan Buchar committed 2 months ago
refactor(core): retire TS-private test seams and convert to native private fields (#4043)
69a824b Harry committed 3 months ago
chore: use typescript 7 for the packages, keep 6 for the docs build (#4058)
3c2dd5c Martin Adámek committed 3 months ago
chore(deps): security bumps + in-range freshening (#4052)
6d77689 apify-deps[bot] committed 3 months ago