Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
chore(deps): update dependency @inquirer/prompts to v8 (#4195)
189a04f renovate[bot] committed 1 month ago
chore(deps): update dependency camoufox-js to ^0.12.0 (#4189)
c9df866 renovate[bot] committed 1 month ago
chore(deps): update actions/checkout action to v7 (#4193)
0b32f4c renovate[bot] committed 1 month ago
chore(deps): update actions/setup-node action to v7 (#4194)
82af0bd renovate[bot] committed 1 month ago
chore(deps): update opentelemetry-js monorepo to ^0.222.0 (#4190)
e846bc8 renovate[bot] committed 1 month ago
fix(utils): load SAX parser from CommonJS default export (#4186)
5d55386 XuJian committed 1 month ago
chore: Replace ts-private with #private in all methods (#4185)
a256d23 Jan Buchar committed 1 month ago
test: serve the requestsFromUrl lists locally in the throttling warning test (#4188)
fc525f1 Mikhail Koviazin committed 1 month ago
fix: replace a retired session in `SessionPool.addSession()` (#4165)
a97a311 Muhammad Haadhee Sheeraz Mian committed 1 month ago
fix(utils): treat a missing robots.txt (HTTP 404) as allow-all (#4171)
f3fd764 breken committed 1 month ago
refactor: Improve the way IBrowserPool.closePage contract receives session errors (#4161)
438e341 Jan Buchar committed 1 month ago
chore: correct keepAlive JSDoc on stop() draining behavior (#4163)
afd4946 Shervin Babaei committed 1 month ago
docs: Update contributing guide to stop teaching yarn (#4178)
e1a9102 Jan Buchar committed 1 month ago
chore(e2e): bump apify SDK to 4.0.0-beta.46 and fix the E2E suite (#4160)
3c73bfd Vlada Dusek committed 1 month ago
chore(deps): update oxlint, oxfmt and @apify/oxlint-config (#4176)
6284c4e Martin Adámek committed 1 month ago
chore(deps): security bumps + minimatch override prune (#4175)
9e36808 apify-deps[bot] committed 1 month ago
chore(deps): bump brace-expansion from 1.1.18 to 1.1.21 in /website/versioned_docs/version-3.17 (#4174)
c0f3bf4 dependabot[bot] committed 1 month ago
fix(playwright): keep detecting a Cloudflare challenge that navigates itself (#4170)
f90b51c Martin Adámek committed 1 month ago
docs: drop v4-only changes from the v3 upgrading guide (#4168)
dd97b4d Martin Adámek committed 1 month ago
fix!: Remove mostly unused Request.userData.forefront field (#4157)
761be27 Jan Buchar committed 1 month ago
fix(http-crawler): forward the context proxy URL to the HTTP client (#4152)
b4e6f21 Jindřich Bär committed 1 month ago