Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore(release): v3.17.0

Apify Release Botcacad22ea8cf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2d1715823..69f0e3a74 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,52 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* add network timeouts to discoverValidSitemaps to prevent indefinite hangs ([#3429](https://github.com/apify/crawlee/issues/3429)) ([ee90835](https://github.com/apify/crawlee/commit/ee90835d2cbbea55815f6e417450b8c3d6b140ce))
+* bind `Browser.pages()` correctly in `PuppeteerPlugin` ([#3439](https://github.com/apify/crawlee/issues/3439)) ([c3a4b3b](https://github.com/apify/crawlee/commit/c3a4b3b0d5be63f1f7a779ff43560ab2b426f3bb))
+* **browser-pool:** allow unsetting launch context proxy URL ([#3621](https://github.com/apify/crawlee/issues/3621)) ([d5ab219](https://github.com/apify/crawlee/commit/d5ab219ec65289a6117c9273a93e2a70dbb5793a))
+* **browser-pool:** preserve caller's AbortContext across p-limit queue ([#3673](https://github.com/apify/crawlee/issues/3673)) ([c4d58d9](https://github.com/apify/crawlee/commit/c4d58d96a53800d48c375f2e0357546a9e754417)), closes [#3670](https://github.com/apify/crawlee/issues/3670)
+* BrowserCrawler closes ctx.page before errorHandler runs on navig… ([#3655](https://github.com/apify/crawlee/issues/3655)) ([ab34886](https://github.com/apify/crawlee/commit/ab348864b31edebb45e2e11f3ded3489131dcc0f))
+* change sequential approach to parallel for Iterator first page ([#3402](https://github.com/apify/crawlee/issues/3402)) ([adf3dae](https://github.com/apify/crawlee/commit/adf3daeb0b3000124817b1b2011b2196342e8715)), closes [#3395](https://github.com/apify/crawlee/issues/3395)
+* **core:** apply object searchParams in HTTP request options ([#3622](https://github.com/apify/crawlee/issues/3622)) ([d32946e](https://github.com/apify/crawlee/commit/d32946efbdc95bf71dfcae67b4509244e1857956))
+* **core:** release storage open queue slot on failure ([#3662](https://github.com/apify/crawlee/issues/3662)) ([18ebc61](https://github.com/apify/crawlee/commit/18ebc61b27263437f6b3814fb774e65c2c4f3b88)), closes [#3661](https://github.com/apify/crawlee/issues/3661)
+* correct failed requests delta count in periodic log message ([#3482](https://github.com/apify/crawlee/issues/3482)) ([a098f42](https://github.com/apify/crawlee/commit/a098f420efdcec4b1827327c012c6eb365460838))
+* Correctly track the number of requests handled by a crawler ([#3410](https://github.com/apify/crawlee/issues/3410)) ([b2296ce](https://github.com/apify/crawlee/commit/b2296cea708fa1e83bfdf20d94067bfa51a2e2ed)), closes [#3083](https://github.com/apify/crawlee/issues/3083) [#3330](https://github.com/apify/crawlee/issues/3330)
+* **e2e:** resolve flaky Camoufox tests due to GitHub rate limits ([#3527](https://github.com/apify/crawlee/issues/3527)) ([08c98cf](https://github.com/apify/crawlee/commit/08c98cf52de5fc227afa97b2ff71c52bbe832ba3)), closes [#3524](https://github.com/apify/crawlee/issues/3524)
+* honor custom respectRobotsTxtFile userAgent in enqueueLinks ([#3578](https://github.com/apify/crawlee/issues/3578)) ([1d2528b](https://github.com/apify/crawlee/commit/1d2528b9a0e13841a0acf918e754d42862e84912))
+* log warning when Sitemap.load returns empty sitemap due to loading error ([#3462](https://github.com/apify/crawlee/issues/3462)) ([2cf9c1c](https://github.com/apify/crawlee/commit/2cf9c1cd9fca21ea5113ee1404dae3a6a652fec0))
+* parse proxy URLs correctly when using `useIncognitoPages` ([#3433](https://github.com/apify/crawlee/issues/3433)) ([db2bb68](https://github.com/apify/crawlee/commit/db2bb6817e0d193471330aeed30d1ead8ecee465))
+* Prevent accidental request dropping with `maxRequestsPerCrawl` ([#3531](https://github.com/apify/crawlee/issues/3531)) ([b23319b](https://github.com/apify/crawlee/commit/b23319bbe8c171541f17f343f718867e25bdb620))
+* prevent iframe expansion failure on pages with Trusted Types CSP ([#3590](https://github.com/apify/crawlee/issues/3590)) ([c0b9b50](https://github.com/apify/crawlee/commit/c0b9b50518bab6e62c62125aeefba6b919046d63)), closes [#3588](https://github.com/apify/crawlee/issues/3588)
+* prevent v3 sitemap discovery init hangs in discoverValidSitemaps ([#3434](https://github.com/apify/crawlee/issues/3434)) ([6c04f92](https://github.com/apify/crawlee/commit/6c04f923ceca6963e25ad73ebd41e480b0d7c0e4)), closes [#3412](https://github.com/apify/crawlee/issues/3412)
+* unref BrowserController.close() fallback timer to avoid keeping Node alive longer than needed ([#3671](https://github.com/apify/crawlee/issues/3671)) ([eef94d4](https://github.com/apify/crawlee/commit/eef94d441503c3ee879f396d86a96a5ac7d0edc8))
+* using GET for candidate checks - sitemaps ([#3464](https://github.com/apify/crawlee/issues/3464)) ([f82eb04](https://github.com/apify/crawlee/commit/f82eb04673b6a9afd0e56dac374a2fa5c91b42e0)), closes [#3463](https://github.com/apify/crawlee/issues/3463)
+
+
+### Features
+
+* **adaptive-crawler:** add shouldPropagateError callback to prevent browser fallback ([#3635](https://github.com/apify/crawlee/issues/3635)) ([d3a29d9](https://github.com/apify/crawlee/commit/d3a29d9623bfd3b9b75c496a1c48134532c824e5))
+* add `ignoreProxyCertificate` option for the internal `proxy-chain` instance ([#3418](https://github.com/apify/crawlee/issues/3418)) ([02eec66](https://github.com/apify/crawlee/commit/02eec66ecfaef94dd9903940d7969e72cf4b1fd1)), closes [#3369](https://github.com/apify/crawlee/issues/3369)
+* add abortDownload context helper to FileDownload ([#3528](https://github.com/apify/crawlee/issues/3528)) ([3b3d8f6](https://github.com/apify/crawlee/commit/3b3d8f685b079dea6594758a5897ba2a3181c3ac))
+* Dynamic memory snapshots ([#3471](https://github.com/apify/crawlee/issues/3471)) ([87c67dc](https://github.com/apify/crawlee/commit/87c67dc4084b11f04cc62926489d657041e60e2c)), closes [#3408](https://github.com/apify/crawlee/issues/3408)
+* expose Download objects on PlaywrightCrawlingContext ([#3596](https://github.com/apify/crawlee/issues/3596)) ([5dc1627](https://github.com/apify/crawlee/commit/5dc1627deccd3d04d12fefec99b0e119fb87cbe4)), closes [#3583](https://github.com/apify/crawlee/issues/3583)
+* Modularize SystemStatus to allow custom backpressure mechanisms ([#3529](https://github.com/apify/crawlee/issues/3529)) ([7159a13](https://github.com/apify/crawlee/commit/7159a13e0fb13661b9d6fa8cae1a0713bdbfb4eb))
+* **templates:** use ImpitHttpClient and install optional deps in Docker ([#3603](https://github.com/apify/crawlee/issues/3603)) ([113dca6](https://github.com/apify/crawlee/commit/113dca699960f2018341ee8342b9b6af70252153)), closes [#3601](https://github.com/apify/crawlee/issues/3601)
+* **utils:** add sitemapFilter option to parseSitemap ([#3557](https://github.com/apify/crawlee/issues/3557)) ([1d4f6b9](https://github.com/apify/crawlee/commit/1d4f6b9ca44b8224f14584c337bd80209eea3819))
+
+
+### Performance Improvements
+
+* add lazyIterablePromise to kvs iterators ([#3413](https://github.com/apify/crawlee/issues/3413)) ([2658610](https://github.com/apify/crawlee/commit/2658610a62ae8dca2702f5af222d64f88d91f4a8))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/lerna.json b/lerna.json
index b7be0a468..b3dafc27e 100644
--- a/lerna.json
+++ b/lerna.json
@@ -2,7 +2,7 @@
"packages": [
"packages/*"
],
- "version": "3.16.0",
+ "version": "3.17.0",
"command": {
"version": {
"conventionalCommits": true,
diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.md
index 446580203..b2defe139 100644
--- a/packages/basic-crawler/CHANGELOG.md
+++ b/packages/basic-crawler/CHANGELOG.md
@@ -3,6 +3,20 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* correct failed requests delta count in periodic log message ([#3482](https://github.com/apify/crawlee/issues/3482)) ([a098f42](https://github.com/apify/crawlee/commit/a098f420efdcec4b1827327c012c6eb365460838))
+* Correctly track the number of requests handled by a crawler ([#3410](https://github.com/apify/crawlee/issues/3410)) ([b2296ce](https://github.com/apify/crawlee/commit/b2296cea708fa1e83bfdf20d94067bfa51a2e2ed)), closes [#3083](https://github.com/apify/crawlee/issues/3083) [#3330](https://github.com/apify/crawlee/issues/3330)
+* honor custom respectRobotsTxtFile userAgent in enqueueLinks ([#3578](https://github.com/apify/crawlee/issues/3578)) ([1d2528b](https://github.com/apify/crawlee/commit/1d2528b9a0e13841a0acf918e754d42862e84912))
+* Prevent accidental request dropping with `maxRequestsPerCrawl` ([#3531](https://github.com/apify/crawlee/issues/3531)) ([b23319b](https://github.com/apify/crawlee/commit/b23319bbe8c171541f17f343f718867e25bdb620))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
index ed1c45e03..e6073cd3a 100644
--- a/packages/basic-crawler/package.json
+++ b/packages/basic-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/basic",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -48,9 +48,9 @@
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/core": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/core": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"csv-stringify": "^6.2.0",
"fs-extra": "^11.0.0",
"got-scraping": "^4.2.1",
diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.md
index cef3a5be4..2c8fd8151 100644
--- a/packages/browser-crawler/CHANGELOG.md
+++ b/packages/browser-crawler/CHANGELOG.md
@@ -3,6 +3,22 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* BrowserCrawler closes ctx.page before errorHandler runs on navig… ([#3655](https://github.com/apify/crawlee/issues/3655)) ([ab34886](https://github.com/apify/crawlee/commit/ab348864b31edebb45e2e11f3ded3489131dcc0f))
+
+
+### Features
+
+* add `ignoreProxyCertificate` option for the internal `proxy-chain` instance ([#3418](https://github.com/apify/crawlee/issues/3418)) ([02eec66](https://github.com/apify/crawlee/commit/02eec66ecfaef94dd9903940d7969e72cf4b1fd1)), closes [#3369](https://github.com/apify/crawlee/issues/3369)
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.json
index dc0a71edb..b5a2b291f 100644
--- a/packages/browser-crawler/package.json
+++ b/packages/browser-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/browser",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -54,10 +54,10 @@
},
"dependencies": {
"@apify/timeout": "^0.3.0",
- "@crawlee/basic": "3.16.0",
- "@crawlee/browser-pool": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/basic": "^3.17.0",
+ "@crawlee/browser-pool": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"ow": "^0.28.1",
"tslib": "^2.4.0",
"type-fest": "^4.0.0"
diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.md
index 8fb148e5a..3cfd1cbd6 100644
--- a/packages/browser-pool/CHANGELOG.md
+++ b/packages/browser-pool/CHANGELOG.md
@@ -3,6 +3,27 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* bind `Browser.pages()` correctly in `PuppeteerPlugin` ([#3439](https://github.com/apify/crawlee/issues/3439)) ([c3a4b3b](https://github.com/apify/crawlee/commit/c3a4b3b0d5be63f1f7a779ff43560ab2b426f3bb))
+* **browser-pool:** allow unsetting launch context proxy URL ([#3621](https://github.com/apify/crawlee/issues/3621)) ([d5ab219](https://github.com/apify/crawlee/commit/d5ab219ec65289a6117c9273a93e2a70dbb5793a))
+* **browser-pool:** preserve caller's AbortContext across p-limit queue ([#3673](https://github.com/apify/crawlee/issues/3673)) ([c4d58d9](https://github.com/apify/crawlee/commit/c4d58d96a53800d48c375f2e0357546a9e754417)), closes [#3670](https://github.com/apify/crawlee/issues/3670)
+* BrowserCrawler closes ctx.page before errorHandler runs on navig… ([#3655](https://github.com/apify/crawlee/issues/3655)) ([ab34886](https://github.com/apify/crawlee/commit/ab348864b31edebb45e2e11f3ded3489131dcc0f))
+* parse proxy URLs correctly when using `useIncognitoPages` ([#3433](https://github.com/apify/crawlee/issues/3433)) ([db2bb68](https://github.com/apify/crawlee/commit/db2bb6817e0d193471330aeed30d1ead8ecee465))
+* unref BrowserController.close() fallback timer to avoid keeping Node alive longer than needed ([#3671](https://github.com/apify/crawlee/issues/3671)) ([eef94d4](https://github.com/apify/crawlee/commit/eef94d441503c3ee879f396d86a96a5ac7d0edc8))
+
+
+### Features
+
+* add `ignoreProxyCertificate` option for the internal `proxy-chain` instance ([#3418](https://github.com/apify/crawlee/issues/3418)) ([02eec66](https://github.com/apify/crawlee/commit/02eec66ecfaef94dd9903940d7969e72cf4b1fd1)), closes [#3369](https://github.com/apify/crawlee/issues/3369)
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.json
index ceb00ea97..fb1b1597b 100644
--- a/packages/browser-pool/package.json
+++ b/packages/browser-pool/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/browser-pool",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -38,8 +38,8 @@
"dependencies": {
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.0",
- "@crawlee/core": "3.16.0",
- "@crawlee/types": "3.16.0",
+ "@crawlee/core": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
"fingerprint-generator": "^2.1.68",
"fingerprint-injector": "^2.1.68",
"lodash.merge": "^4.6.2",
diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.md
index 5fcd2e103..cd19de0ab 100644
--- a/packages/cheerio-crawler/CHANGELOG.md
+++ b/packages/cheerio-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/cheerio
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.json
index 5047d5694..ec12b6693 100644
--- a/packages/cheerio-crawler/package.json
+++ b/packages/cheerio-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/cheerio",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -53,9 +53,9 @@
"access": "public"
},
"dependencies": {
- "@crawlee/http": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/http": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"cheerio": "1.0.0-rc.12",
"htmlparser2": "^9.0.0",
"tslib": "^2.4.0"
diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md
index 79b4e4c14..843c3601d 100644
--- a/packages/cli/CHANGELOG.md
+++ b/packages/cli/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/cli
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/cli/package.json b/packages/cli/package.json
index a92a3f767..80467a480 100644
--- a/packages/cli/package.json
+++ b/packages/cli/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/cli",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -51,7 +51,7 @@
"access": "public"
},
"dependencies": {
- "@crawlee/templates": "3.16.0",
+ "@crawlee/templates": "^3.17.0",
"ansi-colors": "^4.1.3",
"fs-extra": "^11.0.0",
"inquirer": "^8.2.4",
diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.md
index 35c5bf9af..e70f308fb 100644
--- a/packages/core/CHANGELOG.md
+++ b/packages/core/CHANGELOG.md
@@ -3,6 +3,25 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* **core:** release storage open queue slot on failure ([#3662](https://github.com/apify/crawlee/issues/3662)) ([18ebc61](https://github.com/apify/crawlee/commit/18ebc61b27263437f6b3814fb774e65c2c4f3b88)), closes [#3661](https://github.com/apify/crawlee/issues/3661)
+* honor custom respectRobotsTxtFile userAgent in enqueueLinks ([#3578](https://github.com/apify/crawlee/issues/3578)) ([1d2528b](https://github.com/apify/crawlee/commit/1d2528b9a0e13841a0acf918e754d42862e84912))
+* Prevent accidental request dropping with `maxRequestsPerCrawl` ([#3531](https://github.com/apify/crawlee/issues/3531)) ([b23319b](https://github.com/apify/crawlee/commit/b23319bbe8c171541f17f343f718867e25bdb620))
+
+
+### Features
+
+* Dynamic memory snapshots ([#3471](https://github.com/apify/crawlee/issues/3471)) ([87c67dc](https://github.com/apify/crawlee/commit/87c67dc4084b11f04cc62926489d657041e60e2c)), closes [#3408](https://github.com/apify/crawlee/issues/3408)
+* Modularize SystemStatus to allow custom backpressure mechanisms ([#3529](https://github.com/apify/crawlee/issues/3529)) ([7159a13](https://github.com/apify/crawlee/commit/7159a13e0fb13661b9d6fa8cae1a0713bdbfb4eb))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/core/package.json b/packages/core/package.json
index e3290c108..a58f68076 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/core",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -59,9 +59,9 @@
"@apify/pseudo_url": "^2.0.30",
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/memory-storage": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/memory-storage": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"@sapphire/async-queue": "^1.5.1",
"@vladfrangu/async_event_emitter": "^2.2.2",
"csv-stringify": "^6.2.0",
diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.md
index 12109d6c5..b0b99dbeb 100644
--- a/packages/crawlee/CHANGELOG.md
+++ b/packages/crawlee/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package crawlee
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
index 5b9a78f11..18ea7c244 100644
--- a/packages/crawlee/package.json
+++ b/packages/crawlee/package.json
@@ -1,6 +1,6 @@
{
"name": "crawlee",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -54,18 +54,18 @@
"access": "public"
},
"dependencies": {
- "@crawlee/basic": "3.16.0",
- "@crawlee/browser": "3.16.0",
- "@crawlee/browser-pool": "3.16.0",
- "@crawlee/cheerio": "3.16.0",
- "@crawlee/cli": "3.16.0",
- "@crawlee/core": "3.16.0",
- "@crawlee/http": "3.16.0",
- "@crawlee/jsdom": "3.16.0",
- "@crawlee/linkedom": "3.16.0",
- "@crawlee/playwright": "3.16.0",
- "@crawlee/puppeteer": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/basic": "^3.17.0",
+ "@crawlee/browser": "^3.17.0",
+ "@crawlee/browser-pool": "^3.17.0",
+ "@crawlee/cheerio": "^3.17.0",
+ "@crawlee/cli": "^3.17.0",
+ "@crawlee/core": "^3.17.0",
+ "@crawlee/http": "^3.17.0",
+ "@crawlee/jsdom": "^3.17.0",
+ "@crawlee/linkedom": "^3.17.0",
+ "@crawlee/playwright": "^3.17.0",
+ "@crawlee/puppeteer": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"import-local": "^3.1.0",
"tslib": "^2.4.0"
},
diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.md
index f81a26b71..08e455e09 100644
--- a/packages/http-crawler/CHANGELOG.md
+++ b/packages/http-crawler/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Features
+
+* add abortDownload context helper to FileDownload ([#3528](https://github.com/apify/crawlee/issues/3528)) ([3b3d8f6](https://github.com/apify/crawlee/commit/3b3d8f685b079dea6594758a5897ba2a3181c3ac))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.json
index f7317eded..89ce746db 100644
--- a/packages/http-crawler/package.json
+++ b/packages/http-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/http",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,9 +55,9 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/basic": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/basic": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"@types/content-type": "^1.1.5",
"cheerio": "1.0.0-rc.12",
"content-type": "^1.0.4",
diff --git a/packages/impit-client/CHANGELOG.md b/packages/impit-client/CHANGELOG.md
index c01fc0040..25e688f8d 100644
--- a/packages/impit-client/CHANGELOG.md
+++ b/packages/impit-client/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/impit-client
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/impit-client/package.json b/packages/impit-client/package.json
index 904f9e7df..d23cbd897 100644
--- a/packages/impit-client/package.json
+++ b/packages/impit-client/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/impit-client",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "impit-based HTTP client implementation for Crawlee. Impersonates browser requests to avoid bot detection.",
"engines": {
"node": ">=20.0.0"
@@ -56,7 +56,7 @@
"@crawlee/core": "^3.12.1"
},
"devDependencies": {
- "@crawlee/core": "^3.16.0"
+ "@crawlee/core": "^3.17.0"
},
"dependencies": {
"@apify/datastructures": "^2.0.3",
diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.md
index d127c83b2..7f77ca364 100644
--- a/packages/jsdom-crawler/CHANGELOG.md
+++ b/packages/jsdom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/jsdom
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/jsdom-crawler/package.json b/packages/jsdom-crawler/package.json
index a664b2661..1957d0488 100644
--- a/packages/jsdom-crawler/package.json
+++ b/packages/jsdom-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/jsdom",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,9 +55,9 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/http": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/http": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"@types/jsdom": "^21.0.0",
"cheerio": "1.0.0-rc.12",
"jsdom": "^26.0.0",
diff --git a/packages/linkedom-crawler/CHANGELOG.md b/packages/linkedom-crawler/CHANGELOG.md
index 6f4e40a47..14a3c64aa 100644
--- a/packages/linkedom-crawler/CHANGELOG.md
+++ b/packages/linkedom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/linkedom
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/linkedom-crawler/package.json b/packages/linkedom-crawler/package.json
index ca8a4ba28..371ea8d13 100644
--- a/packages/linkedom-crawler/package.json
+++ b/packages/linkedom-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/linkedom",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,8 +55,8 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/http": "3.16.0",
- "@crawlee/types": "3.16.0",
+ "@crawlee/http": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
"linkedom": "^0.18.0",
"ow": "^0.28.2",
"tslib": "^2.4.0"
diff --git a/packages/memory-storage/CHANGELOG.md b/packages/memory-storage/CHANGELOG.md
index 47525fe4d..09055adf4 100644
--- a/packages/memory-storage/CHANGELOG.md
+++ b/packages/memory-storage/CHANGELOG.md
@@ -3,6 +3,22 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* change sequential approach to parallel for Iterator first page ([#3402](https://github.com/apify/crawlee/issues/3402)) ([adf3dae](https://github.com/apify/crawlee/commit/adf3daeb0b3000124817b1b2011b2196342e8715)), closes [#3395](https://github.com/apify/crawlee/issues/3395)
+
+
+### Performance Improvements
+
+* add lazyIterablePromise to kvs iterators ([#3413](https://github.com/apify/crawlee/issues/3413)) ([2658610](https://github.com/apify/crawlee/commit/2658610a62ae8dca2702f5af222d64f88d91f4a8))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/memory-storage/package.json b/packages/memory-storage/package.json
index 22cf852d4..d67558b2c 100644
--- a/packages/memory-storage/package.json
+++ b/packages/memory-storage/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/memory-storage",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "A simple in-memory storage implementation of the Apify API",
"engines": {
"node": ">= 16"
@@ -49,7 +49,7 @@
},
"dependencies": {
"@apify/log": "^2.4.0",
- "@crawlee/types": "3.16.0",
+ "@crawlee/types": "^3.17.0",
"@sapphire/async-queue": "^1.5.0",
"@sapphire/shapeshift": "^3.0.0",
"content-type": "^1.0.4",
diff --git a/packages/playwright-crawler/CHANGELOG.md b/packages/playwright-crawler/CHANGELOG.md
index d16ec6a9f..3b531401f 100644
--- a/packages/playwright-crawler/CHANGELOG.md
+++ b/packages/playwright-crawler/CHANGELOG.md
@@ -3,6 +3,24 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* Prevent accidental request dropping with `maxRequestsPerCrawl` ([#3531](https://github.com/apify/crawlee/issues/3531)) ([b23319b](https://github.com/apify/crawlee/commit/b23319bbe8c171541f17f343f718867e25bdb620))
+* prevent iframe expansion failure on pages with Trusted Types CSP ([#3590](https://github.com/apify/crawlee/issues/3590)) ([c0b9b50](https://github.com/apify/crawlee/commit/c0b9b50518bab6e62c62125aeefba6b919046d63)), closes [#3588](https://github.com/apify/crawlee/issues/3588)
+
+
+### Features
+
+* **adaptive-crawler:** add shouldPropagateError callback to prevent browser fallback ([#3635](https://github.com/apify/crawlee/issues/3635)) ([d3a29d9](https://github.com/apify/crawlee/commit/d3a29d9623bfd3b9b75c496a1c48134532c824e5))
+* expose Download objects on PlaywrightCrawlingContext ([#3596](https://github.com/apify/crawlee/issues/3596)) ([5dc1627](https://github.com/apify/crawlee/commit/5dc1627deccd3d04d12fefec99b0e119fb87cbe4)), closes [#3583](https://github.com/apify/crawlee/issues/3583)
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/playwright-crawler/package.json b/packages/playwright-crawler/package.json
index 2e14429f7..2a96696b5 100644
--- a/packages/playwright-crawler/package.json
+++ b/packages/playwright-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/playwright",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -56,11 +56,11 @@
"@apify/datastructures": "^2.0.0",
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.1",
- "@crawlee/browser": "3.16.0",
- "@crawlee/browser-pool": "3.16.0",
- "@crawlee/core": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/browser": "^3.17.0",
+ "@crawlee/browser-pool": "^3.17.0",
+ "@crawlee/core": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"cheerio": "1.0.0-rc.12",
"jquery": "^3.6.0",
"lodash.isequal": "^4.5.0",
diff --git a/packages/puppeteer-crawler/CHANGELOG.md b/packages/puppeteer-crawler/CHANGELOG.md
index 7170ef35d..47ca8dfbd 100644
--- a/packages/puppeteer-crawler/CHANGELOG.md
+++ b/packages/puppeteer-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/puppeteer
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/puppeteer-crawler/package.json b/packages/puppeteer-crawler/package.json
index 48e46a6c1..002326c21 100644
--- a/packages/puppeteer-crawler/package.json
+++ b/packages/puppeteer-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/puppeteer",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,10 +55,10 @@
"dependencies": {
"@apify/datastructures": "^2.0.0",
"@apify/log": "^2.4.0",
- "@crawlee/browser": "3.16.0",
- "@crawlee/browser-pool": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/browser": "^3.17.0",
+ "@crawlee/browser-pool": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"cheerio": "1.0.0-rc.12",
"devtools-protocol": "*",
"idcac-playwright": "^0.2.0",
diff --git a/packages/stagehand-crawler/CHANGELOG.md b/packages/stagehand-crawler/CHANGELOG.md
index 3ff73144c..9b7ebe12b 100644
--- a/packages/stagehand-crawler/CHANGELOG.md
+++ b/packages/stagehand-crawler/CHANGELOG.md
@@ -3,6 +3,22 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* parse proxy URLs correctly when using `useIncognitoPages` ([#3433](https://github.com/apify/crawlee/issues/3433)) ([db2bb68](https://github.com/apify/crawlee/commit/db2bb6817e0d193471330aeed30d1ead8ecee465))
+
+
+### Features
+
+* add `ignoreProxyCertificate` option for the internal `proxy-chain` instance ([#3418](https://github.com/apify/crawlee/issues/3418)) ([02eec66](https://github.com/apify/crawlee/commit/02eec66ecfaef94dd9903940d7969e72cf4b1fd1)), closes [#3369](https://github.com/apify/crawlee/issues/3369)
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/stagehand-crawler/package.json b/packages/stagehand-crawler/package.json
index 92c8a41a0..533e71dd0 100644
--- a/packages/stagehand-crawler/package.json
+++ b/packages/stagehand-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/stagehand",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "AI-powered web crawling with Stagehand integration for Crawlee - enables natural language browser automation with act(), extract(), and observe() methods.",
"engines": {
"node": ">=16.0.0"
@@ -58,11 +58,11 @@
"dependencies": {
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.1",
- "@crawlee/browser": "3.16.0",
- "@crawlee/browser-pool": "3.16.0",
- "@crawlee/core": "3.16.0",
- "@crawlee/types": "3.16.0",
- "@crawlee/utils": "3.16.0",
+ "@crawlee/browser": "^3.17.0",
+ "@crawlee/browser-pool": "^3.17.0",
+ "@crawlee/core": "^3.17.0",
+ "@crawlee/types": "^3.17.0",
+ "@crawlee/utils": "^3.17.0",
"ow": "^0.28.1",
"tslib": "^2.4.0"
},
diff --git a/packages/templates/CHANGELOG.md b/packages/templates/CHANGELOG.md
index 37722a0a4..bb67c1dd5 100644
--- a/packages/templates/CHANGELOG.md
+++ b/packages/templates/CHANGELOG.md
@@ -3,6 +3,22 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* **e2e:** resolve flaky Camoufox tests due to GitHub rate limits ([#3527](https://github.com/apify/crawlee/issues/3527)) ([08c98cf](https://github.com/apify/crawlee/commit/08c98cf52de5fc227afa97b2ff71c52bbe832ba3)), closes [#3524](https://github.com/apify/crawlee/issues/3524)
+
+
+### Features
+
+* **templates:** use ImpitHttpClient and install optional deps in Docker ([#3603](https://github.com/apify/crawlee/issues/3603)) ([113dca6](https://github.com/apify/crawlee/commit/113dca699960f2018341ee8342b9b6af70252153)), closes [#3601](https://github.com/apify/crawlee/issues/3601)
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/templates/package.json b/packages/templates/package.json
index 098c0b7ed..4635fae35 100644
--- a/packages/templates/package.json
+++ b/packages/templates/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/templates",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "Templates for the crawlee projects",
"engines": {
"node": ">=16.0.0"
diff --git a/packages/types/CHANGELOG.md b/packages/types/CHANGELOG.md
index ec288e390..7bd36c626 100644
--- a/packages/types/CHANGELOG.md
+++ b/packages/types/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+**Note:** Version bump only for package @crawlee/types
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/types/package.json b/packages/types/package.json
index 3dc7849ad..d3e64ddfc 100644
--- a/packages/types/package.json
+++ b/packages/types/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/types",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "Shared types for the crawlee projects",
"engines": {
"node": ">=16.0.0"
diff --git a/packages/utils/CHANGELOG.md b/packages/utils/CHANGELOG.md
index d429f17db..34dc465c5 100644
--- a/packages/utils/CHANGELOG.md
+++ b/packages/utils/CHANGELOG.md
@@ -3,6 +3,27 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.17.0](https://github.com/apify/crawlee/compare/v3.16.0...v3.17.0) (2026-06-04)
+
+
+### Bug Fixes
+
+* add network timeouts to discoverValidSitemaps to prevent indefinite hangs ([#3429](https://github.com/apify/crawlee/issues/3429)) ([ee90835](https://github.com/apify/crawlee/commit/ee90835d2cbbea55815f6e417450b8c3d6b140ce))
+* **core:** apply object searchParams in HTTP request options ([#3622](https://github.com/apify/crawlee/issues/3622)) ([d32946e](https://github.com/apify/crawlee/commit/d32946efbdc95bf71dfcae67b4509244e1857956))
+* log warning when Sitemap.load returns empty sitemap due to loading error ([#3462](https://github.com/apify/crawlee/issues/3462)) ([2cf9c1c](https://github.com/apify/crawlee/commit/2cf9c1cd9fca21ea5113ee1404dae3a6a652fec0))
+* Prevent accidental request dropping with `maxRequestsPerCrawl` ([#3531](https://github.com/apify/crawlee/issues/3531)) ([b23319b](https://github.com/apify/crawlee/commit/b23319bbe8c171541f17f343f718867e25bdb620))
+* prevent v3 sitemap discovery init hangs in discoverValidSitemaps ([#3434](https://github.com/apify/crawlee/issues/3434)) ([6c04f92](https://github.com/apify/crawlee/commit/6c04f923ceca6963e25ad73ebd41e480b0d7c0e4)), closes [#3412](https://github.com/apify/crawlee/issues/3412)
+* using GET for candidate checks - sitemaps ([#3464](https://github.com/apify/crawlee/issues/3464)) ([f82eb04](https://github.com/apify/crawlee/commit/f82eb04673b6a9afd0e56dac374a2fa5c91b42e0)), closes [#3463](https://github.com/apify/crawlee/issues/3463)
+
+
+### Features
+
+* **utils:** add sitemapFilter option to parseSitemap ([#3557](https://github.com/apify/crawlee/issues/3557)) ([1d4f6b9](https://github.com/apify/crawlee/commit/1d4f6b9ca44b8224f14584c337bd80209eea3819))
+
+
+
+
+
# [3.16.0](https://github.com/apify/crawlee/compare/v3.15.3...v3.16.0) (2026-02-06)
diff --git a/packages/utils/package.json b/packages/utils/package.json
index 01826032d..32dab1750 100644
--- a/packages/utils/package.json
+++ b/packages/utils/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/utils",
- "version": "3.16.0",
+ "version": "3.17.0",
"description": "A set of shared utilities that can be used by crawlers",
"engines": {
"node": ">=16.0.0"
@@ -49,7 +49,7 @@
"dependencies": {
"@apify/log": "^2.4.0",
"@apify/ps-tree": "^1.2.0",
- "@crawlee/types": "3.16.0",
+ "@crawlee/types": "^3.17.0",
"@types/sax": "^1.2.7",
"cheerio": "1.0.0-rc.12",
"file-type": "^21.3.1",
diff --git a/yarn.lock b/yarn.lock
index 77e96d121..69304e00b 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -834,16 +834,16 @@ __metadata:
languageName: node
linkType: hard
-"@crawlee/basic@npm:3.16.0, @crawlee/basic@workspace:packages/basic-crawler":
+"@crawlee/basic@npm:^3.17.0, @crawlee/basic@workspace:packages/basic-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/basic@workspace:packages/basic-crawler"
dependencies:
"@apify/log": "npm:^2.4.0"
"@apify/timeout": "npm:^0.3.0"
"@apify/utilities": "npm:^2.7.10"
- "@crawlee/core": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/core": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
csv-stringify: "npm:^6.2.0"
fs-extra: "npm:^11.0.0"
got-scraping: "npm:^4.2.1"
@@ -854,14 +854,14 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/browser-pool@npm:3.16.0, @crawlee/browser-pool@workspace:packages/browser-pool":
+"@crawlee/browser-pool@npm:^3.17.0, @crawlee/browser-pool@workspace:packages/browser-pool":
version: 0.0.0-use.local
resolution: "@crawlee/browser-pool@workspace:packages/browser-pool"
dependencies:
"@apify/log": "npm:^2.4.0"
"@apify/timeout": "npm:^0.3.0"
- "@crawlee/core": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
+ "@crawlee/core": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
fingerprint-generator: "npm:^2.1.68"
fingerprint-injector: "npm:^2.1.68"
lodash.merge: "npm:^4.6.2"
@@ -883,15 +883,15 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/browser@npm:3.16.0, @crawlee/browser@workspace:packages/browser-crawler":
+"@crawlee/browser@npm:^3.17.0, @crawlee/browser@workspace:packages/browser-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/browser@workspace:packages/browser-crawler"
dependencies:
"@apify/timeout": "npm:^0.3.0"
- "@crawlee/basic": "npm:3.16.0"
- "@crawlee/browser-pool": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/basic": "npm:^3.17.0"
+ "@crawlee/browser-pool": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
ow: "npm:^0.28.1"
tslib: "npm:^2.4.0"
type-fest: "npm:^4.0.0"
@@ -906,24 +906,24 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/cheerio@npm:3.16.0, @crawlee/cheerio@workspace:packages/cheerio-crawler":
+"@crawlee/cheerio@npm:^3.17.0, @crawlee/cheerio@workspace:packages/cheerio-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/cheerio@workspace:packages/cheerio-crawler"
dependencies:
- "@crawlee/http": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/http": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
cheerio: "npm:1.0.0-rc.12"
htmlparser2: "npm:^9.0.0"
tslib: "npm:^2.4.0"
languageName: unknown
linkType: soft
-"@crawlee/cli@npm:3.16.0, @crawlee/cli@workspace:packages/cli":
+"@crawlee/cli@npm:^3.17.0, @crawlee/cli@workspace:packages/cli":
version: 0.0.0-use.local
resolution: "@crawlee/cli@workspace:packages/cli"
dependencies:
- "@crawlee/templates": "npm:3.16.0"
+ "@crawlee/templates": "npm:^3.17.0"
ansi-colors: "npm:^4.1.3"
fs-extra: "npm:^11.0.0"
inquirer: "npm:^8.2.4"
@@ -935,7 +935,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/core@npm:3.16.0, @crawlee/core@npm:^3.14.1, @crawlee/core@npm:^3.16.0, @crawlee/core@workspace:packages/core":
+"@crawlee/core@npm:^3.14.1, @crawlee/core@npm:^3.17.0, @crawlee/core@workspace:packages/core":
version: 0.0.0-use.local
resolution: "@crawlee/core@workspace:packages/core"
dependencies:
@@ -945,9 +945,9 @@ __metadata:
"@apify/pseudo_url": "npm:^2.0.30"
"@apify/timeout": "npm:^0.3.0"
"@apify/utilities": "npm:^2.7.10"
- "@crawlee/memory-storage": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/memory-storage": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
"@sapphire/async-queue": "npm:^1.5.1"
"@vladfrangu/async_event_emitter": "npm:^2.2.2"
csv-stringify: "npm:^6.2.0"
@@ -964,15 +964,15 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/http@npm:3.16.0, @crawlee/http@workspace:packages/http-crawler":
+"@crawlee/http@npm:^3.17.0, @crawlee/http@workspace:packages/http-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/http@workspace:packages/http-crawler"
dependencies:
"@apify/timeout": "npm:^0.3.0"
"@apify/utilities": "npm:^2.7.10"
- "@crawlee/basic": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/basic": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
"@types/content-type": "npm:^1.1.5"
cheerio: "npm:1.0.0-rc.12"
content-type: "npm:^1.0.4"
@@ -990,7 +990,7 @@ __metadata:
resolution: "@crawlee/impit-client@workspace:packages/impit-client"
dependencies:
"@apify/datastructures": "npm:^2.0.3"
- "@crawlee/core": "npm:^3.16.0"
+ "@crawlee/core": "npm:^3.17.0"
impit: "npm:^0.14.1"
tough-cookie: "npm:^6.0.0"
peerDependencies:
@@ -998,15 +998,15 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/jsdom@npm:3.16.0, @crawlee/jsdom@workspace:packages/jsdom-crawler":
+"@crawlee/jsdom@npm:^3.17.0, @crawlee/jsdom@workspace:packages/jsdom-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/jsdom@workspace:packages/jsdom-crawler"
dependencies:
"@apify/timeout": "npm:^0.3.0"
"@apify/utilities": "npm:^2.7.10"
- "@crawlee/http": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/http": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
"@types/jsdom": "npm:^21.0.0"
cheerio: "npm:1.0.0-rc.12"
jsdom: "npm:^26.0.0"
@@ -1015,26 +1015,26 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/linkedom@npm:3.16.0, @crawlee/linkedom@workspace:packages/linkedom-crawler":
+"@crawlee/linkedom@npm:^3.17.0, @crawlee/linkedom@workspace:packages/linkedom-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/linkedom@workspace:packages/linkedom-crawler"
dependencies:
"@apify/timeout": "npm:^0.3.0"
"@apify/utilities": "npm:^2.7.10"
- "@crawlee/http": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
+ "@crawlee/http": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
linkedom: "npm:^0.18.0"
ow: "npm:^0.28.2"
tslib: "npm:^2.4.0"
languageName: unknown
linkType: soft
-"@crawlee/memory-storage@npm:3.16.0, @crawlee/memory-storage@workspace:packages/memory-storage":
+"@crawlee/memory-storage@npm:^3.17.0, @crawlee/memory-storage@workspace:packages/memory-storage":
version: 0.0.0-use.local
resolution: "@crawlee/memory-storage@workspace:packages/memory-storage"
dependencies:
"@apify/log": "npm:^2.4.0"
- "@crawlee/types": "npm:3.16.0"
+ "@crawlee/types": "npm:^3.17.0"
"@sapphire/async-queue": "npm:^1.5.0"
"@sapphire/shapeshift": "npm:^3.0.0"
content-type: "npm:^1.0.4"
@@ -1047,18 +1047,18 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/playwright@npm:3.16.0, @crawlee/playwright@workspace:packages/playwright-crawler":
+"@crawlee/playwright@npm:^3.17.0, @crawlee/playwright@workspace:packages/playwright-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/playwright@workspace:packages/playwright-crawler"
dependencies:
"@apify/datastructures": "npm:^2.0.0"
"@apify/log": "npm:^2.4.0"
"@apify/timeout": "npm:^0.3.1"
- "@crawlee/browser": "npm:3.16.0"
- "@crawlee/browser-pool": "npm:3.16.0"
- "@crawlee/core": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/browser": "npm:^3.17.0"
+ "@crawlee/browser-pool": "npm:^3.17.0"
+ "@crawlee/core": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
cheerio: "npm:1.0.0-rc.12"
jquery: "npm:^3.6.0"
lodash.isequal: "npm:^4.5.0"
@@ -1078,16 +1078,16 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/puppeteer@npm:3.16.0, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":
+"@crawlee/puppeteer@npm:^3.17.0, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/puppeteer@workspace:packages/puppeteer-crawler"
dependencies:
"@apify/datastructures": "npm:^2.0.0"
"@apify/log": "npm:^2.4.0"
- "@crawlee/browser": "npm:3.16.0"
- "@crawlee/browser-pool": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/browser": "npm:^3.17.0"
+ "@crawlee/browser-pool": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
cheerio: "npm:1.0.0-rc.12"
devtools-protocol: "npm:*"
idcac-playwright: "npm:^0.2.0"
@@ -1178,11 +1178,11 @@ __metadata:
"@apify/log": "npm:^2.4.0"
"@apify/timeout": "npm:^0.3.1"
"@browserbasehq/stagehand": "npm:3.0.7"
- "@crawlee/browser": "npm:3.16.0"
- "@crawlee/browser-pool": "npm:3.16.0"
- "@crawlee/core": "npm:3.16.0"
- "@crawlee/types": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/browser": "npm:^3.17.0"
+ "@crawlee/browser-pool": "npm:^3.17.0"
+ "@crawlee/core": "npm:^3.17.0"
+ "@crawlee/types": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
ow: "npm:^0.28.1"
playwright: "npm:^1.60.0"
tslib: "npm:^2.4.0"
@@ -1201,7 +1201,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/templates@npm:3.16.0, @crawlee/templates@workspace:packages/templates":
+"@crawlee/templates@npm:^3.17.0, @crawlee/templates@workspace:packages/templates":
version: 0.0.0-use.local
resolution: "@crawlee/templates@workspace:packages/templates"
dependencies:
@@ -1213,7 +1213,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/types@npm:3.16.0, @crawlee/types@npm:^3.14.1, @crawlee/types@npm:^3.3.0, @crawlee/types@workspace:packages/types":
+"@crawlee/types@npm:^3.14.1, @crawlee/types@npm:^3.17.0, @crawlee/types@npm:^3.3.0, @crawlee/types@workspace:packages/types":
version: 0.0.0-use.local
resolution: "@crawlee/types@workspace:packages/types"
dependencies:
@@ -1221,13 +1221,13 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/utils@npm:3.16.0, @crawlee/utils@npm:^3.14.1, @crawlee/utils@workspace:packages/utils":
+"@crawlee/utils@npm:^3.14.1, @crawlee/utils@npm:^3.17.0, @crawlee/utils@workspace:packages/utils":
version: 0.0.0-use.local
resolution: "@crawlee/utils@workspace:packages/utils"
dependencies:
"@apify/log": "npm:^2.4.0"
"@apify/ps-tree": "npm:^1.2.0"
- "@crawlee/types": "npm:3.16.0"
+ "@crawlee/types": "npm:^3.17.0"
"@types/sax": "npm:^1.2.7"
"@types/whatwg-mimetype": "npm:^3.0.2"
cheerio: "npm:1.0.0-rc.12"
@@ -5469,18 +5469,18 @@ __metadata:
version: 0.0.0-use.local
resolution: "crawlee@workspace:packages/crawlee"
dependencies:
- "@crawlee/basic": "npm:3.16.0"
- "@crawlee/browser": "npm:3.16.0"
- "@crawlee/browser-pool": "npm:3.16.0"
- "@crawlee/cheerio": "npm:3.16.0"
- "@crawlee/cli": "npm:3.16.0"
- "@crawlee/core": "npm:3.16.0"
- "@crawlee/http": "npm:3.16.0"
- "@crawlee/jsdom": "npm:3.16.0"
- "@crawlee/linkedom": "npm:3.16.0"
- "@crawlee/playwright": "npm:3.16.0"
- "@crawlee/puppeteer": "npm:3.16.0"
- "@crawlee/utils": "npm:3.16.0"
+ "@crawlee/basic": "npm:^3.17.0"
+ "@crawlee/browser": "npm:^3.17.0"
+ "@crawlee/browser-pool": "npm:^3.17.0"
+ "@crawlee/cheerio": "npm:^3.17.0"
+ "@crawlee/cli": "npm:^3.17.0"
+ "@crawlee/core": "npm:^3.17.0"
+ "@crawlee/http": "npm:^3.17.0"
+ "@crawlee/jsdom": "npm:^3.17.0"
+ "@crawlee/linkedom": "npm:^3.17.0"
+ "@crawlee/playwright": "npm:^3.17.0"
+ "@crawlee/puppeteer": "npm:^3.17.0"
+ "@crawlee/utils": "npm:^3.17.0"
import-local: "npm:^3.1.0"
tslib: "npm:^2.4.0"
peerDependencies: