Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update dependency @apify/docusaurus-plugin-typedoc-api to v5.1.0
f9f1f73 renovate[bot] committed 9 months ago
chore(deps): update patch/minor dependencies to v1.58.1 (#3376)
1990b26 renovate[bot] committed 9 months ago
chore(deps): bump tar from 7.5.4 to 7.5.7 in /website (#3375)
555a5f8 dependabot[bot] committed 9 months ago
chore: bump impit version in `@crawlee/impit-client` (#3374)
5313797 Jindřich Bär committed 9 months ago
refactor(discoverValidSitemaps): check `/sitemap_index.xml` for sitemaps (#3370)
19b0254 Sviatozar Petrenko committed 9 months ago
fix(adaptive-crawler): Fix persistence of rendering type detection results (#3368)
4abca8b Jan Buchar committed 9 months ago
chore(deps): update dependency playwright to v1.58.0 (#3364)
3318147 Martin Adámek committed 9 months ago
feat: add `@crawlee/stagehand` package for AI-powered browser automation (#3331)
a89cb5a Martin Adámek committed 10 months ago
docs: add Impit HTTP client guide to v3.15 docs (#3360)
585bf17 Martin Adámek committed 10 months ago
docs: Update HttpCrawler docs about mimetype handling (#3356)
7a087ae Jan Buchar committed 10 months ago
feat: Add a counter of in-flight rendering type detections (#3355)
565fc34 Jan Buchar committed 10 months ago
docs: update typedoc plugin to v5 and remove rspack resolution (#3353)
9369a1e Martin Adámek committed 10 months ago
feat(utils): add `discoverValidSitemaps` utility (#3339)
29f52ed Sviatozar Petrenko committed 10 months ago
docs: improve Docker image version pinning guidance (#3349)
16a6967 Martin Adámek committed 10 months ago
fix: clean turbo cache and tsbuildinfo files in yarn clean (#3348)
6cd9456 Martin Adámek committed 10 months ago
fix(impit-client): pause fromWeb stream to prevent early consumption (#3347)
72aacb4 nikitachapovskii-dev committed 10 months ago
test: fix flaky keepAlive and isFinishedFunction override tests (#3344)
a978ab3 Martin Adámek committed 10 months ago
test: fix flaky requestHandlerTimeoutSecs test properly (#3343)
049413b Martin Adámek committed 10 months ago
chore(deps): update styfle/cancel-workflow-action action to v0.13.0 (#3342)
e025934 renovate[bot] committed 10 months ago
test: fix flaky requestHandlerTimeoutSecs test (#3340)
5f33f65 Martin Adámek committed 10 months ago
fix(core): ensure `maxCrawlDepth` warning is logged only once (#3337)
9d01334 Martin Adámek committed 10 months ago
chore(deps): update dependency impit to ^0.8.0 (#3332)
c292d3e renovate[bot] committed 10 months ago