Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(test): sitemap backpressure test hangs on cross-hostname URLs (#3885)
d0d8fa1 Jindřich Bär committed 4 months ago
fix(core): unblock backpressured sitemap load on persistState (#3863)
0b31a8b Anas Khan committed 4 months ago
fix: filter sitemap-derived URLs by enqueue strategy (#3797)
9dc3160 Vlada Dusek committed 4 months ago
chore(deps): update dependency @apify/timeout to ^0.4.0 (#3872)
3e45246 renovate[bot] committed 4 months ago
fix(memory-storage): use platform basename in key-value store warning (#3864)
dcefe01 Anas Khan committed 4 months ago
fix(stagehand): wait for Stagehand to register a page before using it (#3865)
238214f Martin Adámek committed 4 months ago
feat: opt-in schema validation of request userData per router label (#3851)
c7347f9 Martin Adámek committed 4 months ago
fix(core): remove `RequestList` persist state listener on `teardown` (#3846)
63394ee Anas Khan committed 4 months ago
fix(utils): advance the CPU tick baseline in systemInfoV2 getCurrentCpuTicks (#3824)
bb23800 Anas Khan committed 4 months ago
fix(core): skip malformed cookie fragments in mergeCookies (#3822)
b3c3ef2 Anas Khan committed 4 months ago
chore(deps): update actions/setup-node action to v7 (#3854)
2c4eef2 renovate[bot] committed 4 months ago
fix(core): clear `AutoscaledPool.pause()` polling interval on timeout (#3847)
42b8914 Anas Khan committed 4 months ago
fix(core): avoid re-submitting already-enqueued requests in addRequestsBatched (#3843)
704c269 nikitachapovskii-dev committed 4 months ago
fix(playwright,puppeteer): only intercept requests for non-GET methods or payloads (#3819)
1942f1e Jindřich Bär committed 4 months ago
chore(deps): update patch/minor dependencies to v3.10.2
07f53af renovate[bot] committed 4 months ago
fix: allow any `clickOptions` in `enqueueLinksByClickingElements` (#3823)
abcbfb1 Anas Khan committed 4 months ago
fix(memory-storage): don't increment `handledRequestCount` when updating a handled request (#3827)
4739d48 Anas Khan committed 4 months ago
docs: fix default values of `maxClientErrors` and `snapshotHistorySecs` (#3828)
15f9e55 Anas Khan committed 4 months ago
fix(basic-crawler): declare `@apify/datastructures` as a dependency (#3815)
4e9d227 Colin McDonnell committed 4 months ago
chore(deps): bump js-yaml from 3.14.2 to 3.15.0 in /website (#3810)
b7b2db8 dependabot[bot] committed 4 months ago
fix: read properties directly from `GotScrapingHttpClient` response (#3807)
c673c8d Jindřich Bär committed 4 months ago