Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(utils): allow apostrophes and asterisks in extracted URLs (#3790)
e9baffd Anas Khan committed 4 months ago
fix(utils): prevent sitemap metadata leak across url without loc (#3795)
895ed62 Anas Khan committed 4 months ago
fix(utils): use article:* selectors for article Open Graph properties (#3789)
b660506 Anas Khan committed 4 months ago
test(e2e): use plain ApifyClient for platform result reads (#3776)
652b926 Martin Adámek committed 4 months ago
test(e2e): de-flake puppeteer-ignore-ssl-errors (#3791)
53b0709 Martin Adámek committed 4 months ago
chore(deps): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /website (#3778)
6a9eab7 dependabot[bot] committed 4 months ago
ci: use apify/actions/pr-title-check wrapper for PR title checks (#3777)
3cb75ea Jan Buchar committed 4 months ago
feat: type-safe router labels via per-label userData map (#3747)
1124aca Martin Adámek committed 5 months ago
fix(core): cap retries for unprocessed requests in addRequestsBatched (#3765)
b3170a6 Martin Adámek committed 5 months ago
feat(impit-client): add `cacheClients` option to disable `Impit` client reuse (#3771)
af9cb8c Copilot committed 5 months ago
chore(deps): update patch/minor dependencies to v5.1.14
fef5685 renovate[bot] committed 5 months ago
docs: Building a Gradcracker scraper with Crawlee blogpost (#3734)
15e77ef Saurav Jain committed 5 months ago
chore(deps): update patch/minor dependencies to v5.1.13
778d12d renovate[bot] committed 5 months ago
chore(deps): bump dompurify from 3.4.10 to 3.4.11 in /website (#3766)
9b93877 dependabot[bot] committed 5 months ago
chore(deps): update actions/checkout action to v7 (#3763)
3765877 renovate[bot] committed 5 months ago
chore(deps): update dependency @apify/tsconfig to ^0.2.0 (#3760)
1fe3e7c renovate[bot] committed 5 months ago
chore(deps): update patch/minor dependencies to v24.17.0
c16f556 renovate[bot] committed 5 months ago
fix: make crawler terminal status message reliably delivered (#3733)
7aed264 Lukáš Křivka committed 5 months ago
chore(deps): bump vulnerable transitive deps to resolve security alerts (#3755)
ac00720 Martin Adámek committed 5 months ago
chore: camoufox-cloudflare e2e + camoufox-ts template on Playwright 1.60 (#3749)
0483e17 Martin Adámek committed 5 months ago
chore(deps): pin eslint tooling and freeze it in renovate (#3746)
78f888d Martin Adámek committed 5 months ago
perf(core): improve `RequestList` initialization speed on large arrays (#3754)
c451c13 Rayan Salhab committed 5 months ago
chore(deps): update patch/minor dependencies to v1.61.0 (#3742)
361596d renovate[bot] committed 5 months ago
chore(deps): bump hono from 4.12.23 to 4.12.25 (#3753)
9784ced dependabot[bot] committed 5 months ago