Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): bump nanoid from 5.1.9 to 5.1.16 (#4046)
9918489 dependabot[bot] committed 3 months ago
docs: add a playbook for switching master to the next major (#4049)
452afb1 Martin Adámek committed 3 months ago
chore(deps): update dependency puppeteer to v25.8.0 (#4048)
b04de35 Martin Adámek committed 3 months ago
chore(ci): point the release workflows at master for the v4 line
2137c70 Martin Adámek committed 3 months ago
chore: restore the typed schema-router overloads on createFileRouter
a1512cf Martin Adámek committed 3 months ago
docs: drop the 4.0 snapshot in favor of serving v4 docs as the next version
2766e87 Martin Adámek committed 3 months ago
chore: Add oxlint plugin to warn about TS-private class members (#4035)
80d7010 Jan Buchar committed 3 months ago
chore(e2e): retry the platform statistics lookup instead of crashing on a read-after-write race
6188678 Martin Adámek committed 3 months ago
docs: warn against --omit=optional in v4 upgrading guide (#4034)
ffd240d Jindřich Bär committed 3 months ago
feat: Add Symbol.asyncDispose to disposable services for DX (#4032)
a7b2f1c Jan Buchar committed 3 months ago
chore(e2e): keep 403 responses retryable in the camoufox-cloudflare fixture
7aea1f5 Martin Adámek committed 3 months ago
docs: pin 3.18 API links in content carried over from master
af00b2a Martin Adámek committed 3 months ago
chore(e2e): pair camoufox-js ^0.11.0 with playwright 1.60.0 in the cloudflare fixture
bcc97ff Martin Adámek committed 3 months ago
chore(e2e): repair E2E suite for the enqueueLinks split and the zod catalog dependency
5a7d0b9 Martin Adámek committed 3 months ago
chore: fix formatting and regenerate public API reports after the rebase
c324386 Martin Adámek committed 3 months ago
chore(test): use crawler.statistics in the maxRequestsPerCrawl budget tests
4bca1e6 Martin Adámek committed 3 months ago
refactor!: Rename BasicCrawler.stats to statistics (#4028)
41a2584 Jan Buchar committed 3 months ago
refactor!: replace `browserPoolOptions` with browser pool factories (#4026)
39148e6 Jan Buchar committed 3 months ago
chore: re-apply master-side fixes over the enqueueLinks split (#4010)
839a9b2 Martin Adámek committed 3 months ago
feat: Accept custom extensions in the IStatistics interface (#3982)
e00e8ce Jan Buchar committed 3 months ago
feat: Improve sameDomainDelaySecs implementation via ThrottlingRequestManager (#4017)
d50177c Jan Buchar committed 3 months ago
chore: Rename client load signal to storageBackend (#4023)
8e3194b Jan Buchar committed 3 months ago
feat: split `enqueueLinks` into `extractLinks` + `addRequests` (#4010)
6bcd06e Jindřich Bär committed 3 months ago
chore(test): finish adapting master-carried tests to v4 APIs
375b08c Martin Adámek committed 3 months ago
chore(test): adapt master-carried tests to v4 APIs after rebase
75412fc Martin Adámek committed 3 months ago
chore(utils): re-add tldts dependency for enqueue-strategy URL filtering
bd20a49 Martin Adámek committed 3 months ago
chore: restore @ts-ignore on robotsParser calls under nodenext resolution
bbd72d8 Martin Adámek committed 3 months ago
chore(templates): keep camoufox-js ^0.11.0 paired with playwright 1.60.0 in the camoufox template
3ce18ea Martin Adámek committed 3 months ago