Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Commits on master
fix(utils): drop query string and fragment from guessed sitemap urls (#4132)
91291b2 Suliman Abdulrazzaq committed 1 month ago
feat: Fully-fledged support for custom (de)serializers in RecoverableState (#4131)
d9c5722 Jan Buchar committed 1 month ago
fix(http-client): do not reuse cookies of the previous hop on same-origin redirects (#4154)
beaf123 Jindřich Bär committed 1 month ago
refactor: Fix tests that depended on private methods (#4150)
54f744e Jan Buchar committed 1 month ago
refactor!: drop the unused `request` option from `ProxyConfiguration` (#4153)
b668452 Jindřich Bär committed 1 month ago
fix(http-client): replay the request body on redirects without throwing (#4136)
3d48f47 Suliman Abdulrazzaq committed 1 month ago
fix(http-client): do not forward credentials across cross-origin redirects (#4137)
f13d499 Suliman Abdulrazzaq committed 1 month ago
refactor!: keep crawler vocabulary out of @crawlee/core's Request (#4148)
8beb96e Jan Buchar committed 1 month ago
feat: Pass task identifiers from AutoscaledPool to IConcurrencySystem (#4130)
7eb0881 Jan Buchar committed 1 month ago
refactor!: remove persistState from IRequestLoader interface (#4042)
4bc80c4 Harry committed 2 months ago
feat(fs-storage): adopt out-of-band key-value files as regular records (#4128)
ecc6ea7 Jan Buchar committed 2 months ago
chore(deps): Update locked Apify SDK version so that e2e tests pass (#4126)
df6ba45 Jan Buchar committed 2 months ago
perf!: import only cheerio/slim in cheerio/http crawlers and utils (#4114)
6864be3 Lukáš Křivka committed 2 months ago
fix(fs-storage)!: only probe bare INPUT and INPUT.json for run input (#4120)
0b2ac45 Vlad Frangu committed 2 months ago
docs(blog): add post on Playwright stealth and fingerprint spoofing (#4115)
fbaed52 Federico committed 2 months ago
docs: document slim Docker images and package manager setup (#4082)
be4a0bf Vlad Frangu committed 2 months ago
feat!: remove `@crawlee/jsdom` and `@crawlee/linkedom` from the monorepo (#4119)
bc11f33 Jindřich Bär committed 2 months ago
fix(fs-storage): honor the configured input key for bare-file reads and purge (#4118)
10a49f1 Vlad Frangu committed 2 months ago
refactor(core)!: move crawler-only code from `@crawlee/core` to `@crawlee/basic` (#4117)
00eac95 Martin Adámek committed 2 months ago
fix(browser-pool): survive a page.close() that never settles (#4116)
3924163 nikitachapovskii-dev committed 2 months ago
feat(http): add DomCrawler with a pluggable DOM parser (#4104)
3f6f7bb Jindřich Bär committed 2 months ago
refactor: Extract per-run crawler state into a CrawlerRun class (#4111)
f6a083f Jan Buchar committed 2 months ago
chore(deps): update @apify/* packages to the new majors (#4106)
bc345b9 Martin Adámek committed 2 months ago