Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
feat(templates): use ImpitHttpClient and install optional deps in Docker (#3603)
113dca6 Jindřich Bär committed 6 months ago
chore(deps): bump postcss from 8.5.6 to 8.5.12 in /website (#3612)
5c0ddf3 dependabot[bot] committed 6 months ago
chore(e2e): bump `apify-node-curl-impersonate` in tests (#3609)
2b4a28f Jindřich Bär committed 6 months ago
feat: expose Download objects on PlaywrightCrawlingContext (#3596)
5dc1627 nikitachapovskii-dev committed 6 months ago
fix: honor custom respectRobotsTxtFile userAgent in enqueueLinks (#3578)
1d2528b Junghwan committed 7 months ago
fix: prevent iframe expansion failure on pages with Trusted Types CSP (#3590)
c0b9b50 nikitachapovskii-dev committed 7 months ago
docs: fix spelling for System Information V2 experiment (#3580)
fc39e1a TomasR committed 7 months ago
chore(deps): bump follow-redirects from 1.15.11 to 1.16.0 in /website (#3573)
e07bd30 dependabot[bot] committed 7 months ago
chore(deps): bump hono from 4.12.12 to 4.12.14 (#3575)
f15ec5c dependabot[bot] committed 7 months ago
chore(deps): bump dompurify from 3.3.3 to 3.4.0 in /website (#3574)
c25d4f7 dependabot[bot] committed 7 months ago
feat(utils): add sitemapFilter option to parseSitemap (#3557)
1d4f6b9 Stefan B. committed 7 months ago
fix: Prevent accidental request dropping with `maxRequestsPerCrawl` (#3531)
b23319b Jan Buchar committed 7 months ago
chore(deps): bump @apify/eslint-config to ^2.0.0 (#3562)
facd4b3 Martin Adámek committed 7 months ago
chore(deps): bump basic-ftp from 5.2.0 to 5.2.2 (#3563)
9655f4c dependabot[bot] committed 7 months ago
chore(deps): bump axios from 1.13.5 to 1.15.0 in /website (#3565)
88fab22 dependabot[bot] committed 7 months ago
chore(deps): bump axios from 1.13.2 to 1.15.0 (#3564)
79b6e79 dependabot[bot] committed 7 months ago
chore: Bump remaining Playwright 1.58.2 references across workspace, templates, E2E actors, and Node 24 Docker images (#3538)
6785cc8 Copilot committed 7 months ago
chore(deps): update patch/minor dependencies (playwright 1.58.2, node 24.14.1) (#3391)
cbef2de renovate[bot] committed 7 months ago
chore(deps): bump lodash from 4.17.23 to 4.18.1 (#3560)
be9da3e dependabot[bot] committed 7 months ago
chore(deps): update apify/actor-node-playwright-camoufox docker tag to v24 (#3530)
3f0eb63 renovate[bot] committed 7 months ago
chore(deps): bump lodash from 4.17.23 to 4.18.1 in /website (#3559)
3d392d6 dependabot[bot] committed 7 months ago
chore(deps): bump @hono/node-server from 1.19.10 to 1.19.13 (#3555)
a7335ed dependabot[bot] committed 7 months ago
chore(deps): bump hono from 4.12.7 to 4.12.12 (#3556)
f966456 dependabot[bot] committed 7 months ago
feat: Modularize SystemStatus to allow custom backpressure mechanisms (#3529)
7159a13 Jan Buchar committed 7 months ago
ci: use standalone apify-cli binary in E2E tests (#3543)
be6dbc4 Martin Adámek committed 7 months ago
docs: serve markdown via Accept header in nginx (#3542)
9586a6a Martin Adámek committed 7 months ago
chore(deps): bump handlebars from 4.7.8 to 4.7.9 (#3539)
a2d34a4 dependabot[bot] committed 7 months ago