Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix: don't give up on a stalled domain while another one is cooling down (#4325)
d10a420 SwastikTripathi committed 14 days ago
fix: Reset consecutive http 429 count on crawl progress (#4354)
ddd9491 Jan Buchar committed 14 days ago
chore(deps): update dependency devtools-protocol to ^0.0.1734725 (#4327)
529e8f2 renovate[bot] committed 15 days ago
test(basic-crawler): pin that minConcurrency tasks keep running under overload (#4316)
11562c8 Vlad Frangu committed 15 days ago
ci: allow Chromium sandbox on the ubuntu-26.04 runners (#4343)
3169dd2 Martin Adámek committed 15 days ago
fix: detect the AWS WAF challenge and CAPTCHA with `retryOnBlocked` (#4184)
eda7b94 Mikhail Koviazin committed 15 days ago
test: assert rejected nested sitemaps are never fetched (#4254)
ec3aa36 Aleksey Safonov committed 15 days ago
feat(basic): pass onCleanup to extendContext callbacks (#4293)
a85394e Junghwan committed 15 days ago
fix(utils): keep urls from the other sitemaps when one sitemap is malformed (#4249)
9e6d897 Jay Hemnani committed 15 days ago
fix(utils): parse gzipped sitemaps with no inner file extension (#4330)
3ecbb9a SwastikTripathi committed 15 days ago
fix(utils): count a microdata property once when itemref reaches it again (#4344)
cf98a4a SwastikTripathi committed 15 days ago
fix: use one enqueue strategy implementation for enqueueLinks, sitemaps and robots.txt (#4345)
b98c573 SwastikTripathi committed 15 days ago
fix(templates): depend on crawlee v4 and fix TypeScript 6 builds (#4338)
a135a8b Jindřich Bär committed 16 days ago
fix(basic): forward `proxyUrl` from `sendRequest` to the HTTP client (#4267)
60c131e Jay Hemnani committed 16 days ago
fix(utils): keep sitemap.txt lines apart when a chunk ends on a line break (#4329)
d01f814 SwastikTripathi committed 16 days ago
fix(core): log background request-batch failures instead of swallowing them (#4103)
c703230 florinBC89 committed 16 days ago
fix(core): make skipNavigation request bookkeeping proxy-safe (#4099)
73ee995 Noah committed 16 days ago
feat(basic-crawler): update request retry statistics in real-time (#3773)
b0237d1 Harry committed 16 days ago
fix(utils): read mailto links with a query or several recipients (#4335)
b18c587 SwastikTripathi committed 17 days ago
fix(utils): accept tel links with a one-digit country code, parameters or percent-encoding (#4336)
69487ea SwastikTripathi committed 17 days ago
fix(adaptive-crawler): Unify deferred cleanup order with BasicCrawler (#4328)
8bdd6b0 Jan Buchar committed 17 days ago
fix(playwright): keep repeated headers in the adaptive browser response (#4321)
c95035a Luigi Ruocco committed 17 days ago
fix(core): give each request its own copy of userData (#4319)
270c713 Nikita Chapovskii committed 17 days ago
perf(utils): make htmlToText linear in the number of nodes (#4322)
4030aa9 Martin Adámek committed 17 days ago
chore(ci): make npm publish survive slow Rekor (sigstore 409 + 30s timeout) (#4323)
a1508fb Martin Adámek committed 17 days ago
chore(deps): dependency sweep — lockfile freshening (#4326)
63fd994 apify-deps[bot] committed 18 days ago