Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
ci: pin `@puppeteer/browsers` to v3 to fix Chrome install on newer Node (#3679)
6af119b Martin Adámek committed 5 months ago
chore(deps): bump lockfile transitives to clear Dependabot alerts (#3678)
46df390 Martin Adámek committed 5 months ago
chore(deps): bump tmp from 0.2.5 to 0.2.6 in /website (#3676)
0a88e1a dependabot[bot] committed 5 months ago
chore(e2e): stream worker logs live and show test-start marker (#3675)
8374db8 Martin Adámek committed 5 months ago
test: make enqueue-links-base e2e tests hermetic (#3674)
3773845 Martin Adámek committed 5 months ago
fix(browser-pool): preserve caller's AbortContext across p-limit queue (#3673)
c4d58d9 Jindřich Bär committed 5 months ago
fix: unref BrowserController.close() fallback timer to avoid keeping Node alive longer than needed (#3671)
eef94d4 Shaun committed 5 months ago
fix(core): release storage open queue slot on failure (#3662)
18ebc61 Jindřich Bär committed 6 months ago
chore(deps): bump webpack-dev-server from 5.2.3 to 5.2.4 in /website (#3659)
f07fda2 dependabot[bot] committed 6 months ago
chore(deps): bump mermaid from 11.12.2 to 11.15.0 in /website (#3658)
a81ccb8 dependabot[bot] committed 6 months ago
chore: Replace calling reusable actions from `apify/workflows/...` to `apify/actions/...` [internal] (#3657)
038ae92 František Nesveda committed 6 months ago
chore(templates): bump actor base images to Node 24 (#3654)
acf4490 Martin Adámek committed 6 months ago
chore(e2e): bump actor base images from Node 20 to Node 24 (#3653)
cd4a748 Martin Adámek committed 6 months ago
feat(adaptive-crawler): add shouldPropagateError callback to prevent browser fallback (#3635)
d3a29d9 Luigi Ruocco committed 6 months ago
chore(deps): bump ip-address from 10.1.0 to 10.2.0 in /website (#3632)
ca72151 dependabot[bot] committed 6 months ago
chore(deps): bump @babel/plugin-transform-modules-systemjs from 7.28.5 to 7.29.4 in /website (#3640)
16a24b5 dependabot[bot] committed 6 months ago
chore(deps): bump fast-uri from 3.1.0 to 3.1.2 (#3638)
10e8e28 dependabot[bot] committed 6 months ago
chore(deps): bump fast-uri from 3.1.0 to 3.1.2 in /website (#3637)
c599424 dependabot[bot] committed 6 months ago
chore(deps): bump hono from 4.12.14 to 4.12.18 (#3639)
440e0f5 dependabot[bot] committed 6 months ago
chore(deps): bump axios from 1.15.0 to 1.16.0 in /website (#3626)
a68102a dependabot[bot] committed 6 months ago
chore(deps): bump axios from 1.15.0 to 1.16.0 (#3625)
594bc1b dependabot[bot] committed 6 months ago
ci: bump @apify/storage-local deps in e2e tests (#3604)
4dca10a Richard Solar committed 6 months ago
fix(browser-pool): allow unsetting launch context proxy URL (#3621)
d5ab219 Adarsh Tiwari committed 6 months ago
fix(core): apply object searchParams in HTTP request options (#3622)
d32946e Adarsh Tiwari committed 6 months ago
chore(templates): add `private: true` to all project template package.json files (#3616)
1d89062 Copilot committed 6 months ago
chore(deps): bump follow-redirects from 1.15.11 to 1.16.0 (#3613)
7deeb03 dependabot[bot] committed 6 months ago
chore(deps): bump postcss from 8.5.6 to 8.5.12 (#3619)
30a1e78 dependabot[bot] committed 6 months ago
ci: rebase before pushing release/version-docs commits (#3614)
3644066 Martin Adámek committed 6 months ago