Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore(release): v3.5.0
Apify Release Bot58cc3d4a74ee
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 395df35cb..1f06707e4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,6 +3,28 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Bug Fixes++* cleanup worker stuff from memory storage to fix `vitest` ([#2004](https://github.com/apify/crawlee/issues/2004)) ([d2e098c](https://github.com/apify/crawlee/commit/d2e098cab62c700a5c58fcf43a5bcf9f492d71ec)), closes [#1999](https://github.com/apify/crawlee/issues/1999)+* **core:** add requests from URL list (`requestsFromUrl`) to the queue in batches ([418fbf8](https://github.com/apify/crawlee/commit/418fbf89d8680f8c460e37cfbf3e521f45770eb2)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+* **core:** support relative links in `enqueueLinks` explicitly provided via `urls` option ([#2014](https://github.com/apify/crawlee/issues/2014)) ([cbd9d08](https://github.com/apify/crawlee/commit/cbd9d08065694b8c86e32c773875cecd41e5fcc9)), closes [#2005](https://github.com/apify/crawlee/issues/2005)+++### Features++* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))+* add support for `sameDomainDelay` ([#2003](https://github.com/apify/crawlee/issues/2003)) ([e796883](https://github.com/apify/crawlee/commit/e79688324790e5d07fc11192769cf051617e96e4)), closes [#1993](https://github.com/apify/crawlee/issues/1993)+* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/lerna.json b/lerna.jsonindex be7d39a90..4e96cb30c 100644--- a/lerna.json+++ b/lerna.json@@ -2,7 +2,7 @@"packages": ["packages/*"],- "version": "3.4.2",+ "version": "3.5.0","command": {"version": {"conventionalCommits": true,diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.mdindex b8dda4ffa..25131bb75 100644--- a/packages/basic-crawler/CHANGELOG.md+++ b/packages/basic-crawler/CHANGELOG.md@@ -3,6 +3,19 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* add support for `sameDomainDelay` ([#2003](https://github.com/apify/crawlee/issues/2003)) ([e796883](https://github.com/apify/crawlee/commit/e79688324790e5d07fc11192769cf051617e96e4)), closes [#1993](https://github.com/apify/crawlee/issues/1993)+* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.jsonindex 210381354..d1ed6245f 100644--- a/packages/basic-crawler/package.json+++ b/packages/basic-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/basic",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -48,9 +48,9 @@"@apify/log": "^2.4.0","@apify/timeout": "^0.3.0","@apify/utilities": "^2.7.10",- "@crawlee/core": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/core": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","got-scraping": "^3.2.9","ow": "^0.28.1","tldts": "^6.0.0",diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.mdindex 6dd6db148..c26f71e75 100644--- a/packages/browser-crawler/CHANGELOG.md+++ b/packages/browser-crawler/CHANGELOG.md@@ -3,6 +3,17 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.jsonindex 1d30a3165..3d2b87d8e 100644--- a/packages/browser-crawler/package.json+++ b/packages/browser-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/browser",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -54,10 +54,10 @@},"dependencies": {"@apify/timeout": "^0.3.0",- "@crawlee/basic": "^3.4.2",- "@crawlee/browser-pool": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/basic": "^3.5.0",+ "@crawlee/browser-pool": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","ow": "^0.28.1","tslib": "^2.4.0"}diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.mdindex 4b6cfb65d..5387728d9 100644--- a/packages/browser-pool/CHANGELOG.md+++ b/packages/browser-pool/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/browser-pool+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/browser-pooldiff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.jsonindex 79daed9bc..3a31bd05e 100644--- a/packages/browser-pool/package.json+++ b/packages/browser-pool/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/browser-pool",- "version": "3.4.2",+ "version": "3.5.0","description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.","engines": {"node": ">=16.0.0"@@ -38,7 +38,7 @@"dependencies": {"@apify/log": "^2.4.0","@apify/timeout": "^0.3.0",- "@crawlee/types": "^3.4.2",+ "@crawlee/types": "^3.5.0","fingerprint-generator": "^2.0.6","fingerprint-injector": "^2.0.5","lodash.merge": "^4.6.2",diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.mdindex a9bd95cb1..d3fbc0299 100644--- a/packages/cheerio-crawler/CHANGELOG.md+++ b/packages/cheerio-crawler/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/cheerio+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/cheeriodiff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.jsonindex 75f3a2e20..871a90bf6 100644--- a/packages/cheerio-crawler/package.json+++ b/packages/cheerio-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/cheerio",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -53,8 +53,8 @@"access": "public"},"dependencies": {- "@crawlee/http": "^3.4.2",- "@crawlee/types": "^3.4.2",+ "@crawlee/http": "^3.5.0",+ "@crawlee/types": "^3.5.0","cheerio": "^1.0.0-rc.12","htmlparser2": "^9.0.0","tslib": "^2.4.0"diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.mdindex 74facbb9a..c8d21ee2a 100644--- a/packages/cli/CHANGELOG.md+++ b/packages/cli/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/cli+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/clidiff --git a/packages/cli/package.json b/packages/cli/package.jsonindex d91b5986d..80646b56f 100644--- a/packages/cli/package.json+++ b/packages/cli/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/cli",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -51,7 +51,7 @@"access": "public"},"dependencies": {- "@crawlee/templates": "^3.4.2",+ "@crawlee/templates": "^3.5.0","ansi-colors": "^4.1.3","fs-extra": "^11.0.0","inquirer": "^8.2.4",diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.mdindex 2303ded86..9abb93eef 100644--- a/packages/core/CHANGELOG.md+++ b/packages/core/CHANGELOG.md@@ -3,6 +3,24 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Bug Fixes++* **core:** add requests from URL list (`requestsFromUrl`) to the queue in batches ([418fbf8](https://github.com/apify/crawlee/commit/418fbf89d8680f8c460e37cfbf3e521f45770eb2)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+* **core:** support relative links in `enqueueLinks` explicitly provided via `urls` option ([#2014](https://github.com/apify/crawlee/issues/2014)) ([cbd9d08](https://github.com/apify/crawlee/commit/cbd9d08065694b8c86e32c773875cecd41e5fcc9)), closes [#2005](https://github.com/apify/crawlee/issues/2005)+++### Features++* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/packages/core/package.json b/packages/core/package.jsonindex 33d42133d..026fe4722 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/core",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -59,9 +59,9 @@"@apify/pseudo_url": "^2.0.30","@apify/timeout": "^0.3.0","@apify/utilities": "^2.7.10",- "@crawlee/memory-storage": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/memory-storage": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","@sapphire/async-queue": "^1.5.0","@types/tough-cookie": "^4.0.2","@vladfrangu/async_event_emitter": "^2.2.2",diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.mdindex 39cb8f60e..f8a8d46cf 100644--- a/packages/crawlee/CHANGELOG.md+++ b/packages/crawlee/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package crawlee+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package crawleediff --git a/packages/crawlee/package.json b/packages/crawlee/package.jsonindex 5bb31a289..9b222aa86 100644--- a/packages/crawlee/package.json+++ b/packages/crawlee/package.json@@ -1,6 +1,6 @@{"name": "crawlee",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -54,17 +54,17 @@"access": "public"},"dependencies": {- "@crawlee/basic": "^3.4.2",- "@crawlee/browser": "^3.4.2",- "@crawlee/cheerio": "^3.4.2",- "@crawlee/cli": "^3.4.2",- "@crawlee/core": "^3.4.2",- "@crawlee/http": "^3.4.2",- "@crawlee/jsdom": "^3.4.2",- "@crawlee/linkedom": "^3.4.2",- "@crawlee/playwright": "^3.4.2",- "@crawlee/puppeteer": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/basic": "^3.5.0",+ "@crawlee/browser": "^3.5.0",+ "@crawlee/cheerio": "^3.5.0",+ "@crawlee/cli": "^3.5.0",+ "@crawlee/core": "^3.5.0",+ "@crawlee/http": "^3.5.0",+ "@crawlee/jsdom": "^3.5.0",+ "@crawlee/linkedom": "^3.5.0",+ "@crawlee/playwright": "^3.5.0",+ "@crawlee/puppeteer": "^3.5.0",+ "@crawlee/utils": "^3.5.0","import-local": "^3.1.0","tslib": "^2.4.0"},diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.mdindex c61d8c054..616e8c18c 100644--- a/packages/http-crawler/CHANGELOG.md+++ b/packages/http-crawler/CHANGELOG.md@@ -3,6 +3,17 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.jsonindex c026d02fb..4f6eab7be 100644--- a/packages/http-crawler/package.json+++ b/packages/http-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/http",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -55,9 +55,9 @@"dependencies": {"@apify/timeout": "^0.3.0","@apify/utilities": "^2.7.10",- "@crawlee/basic": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/basic": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","@types/content-type": "^1.1.5","cheerio": "^1.0.0-rc.12","content-type": "^1.0.4",diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.mdindex fc94018bf..9c4382f63 100644--- a/packages/jsdom-crawler/CHANGELOG.md+++ b/packages/jsdom-crawler/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/jsdom+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/jsdomdiff --git a/packages/jsdom-crawler/package.json b/packages/jsdom-crawler/package.jsonindex 335ee4e09..f782505a6 100644--- a/packages/jsdom-crawler/package.json+++ b/packages/jsdom-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/jsdom",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -55,8 +55,8 @@"dependencies": {"@apify/timeout": "^0.3.0","@apify/utilities": "^2.7.10",- "@crawlee/http": "^3.4.2",- "@crawlee/types": "^3.4.2",+ "@crawlee/http": "^3.5.0",+ "@crawlee/types": "^3.5.0","@types/jsdom": "^21.0.0","cheerio": "^1.0.0-rc.12","jsdom": "^22.0.0",diff --git a/packages/linkedom-crawler/CHANGELOG.md b/packages/linkedom-crawler/CHANGELOG.mdindex 6184c64c9..423809bad 100644--- a/packages/linkedom-crawler/CHANGELOG.md+++ b/packages/linkedom-crawler/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/linkedom+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/linkedomdiff --git a/packages/linkedom-crawler/package.json b/packages/linkedom-crawler/package.jsonindex 3864a79d5..5bd921d23 100644--- a/packages/linkedom-crawler/package.json+++ b/packages/linkedom-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/linkedom",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -55,8 +55,8 @@"dependencies": {"@apify/timeout": "^0.3.0","@apify/utilities": "^2.7.10",- "@crawlee/http": "^3.4.2",- "@crawlee/types": "^3.4.2",+ "@crawlee/http": "^3.5.0",+ "@crawlee/types": "^3.5.0","linkedom": "^0.15.0","ow": "^0.28.2","tslib": "^2.4.0"diff --git a/packages/memory-storage/CHANGELOG.md b/packages/memory-storage/CHANGELOG.mdindex 30462526b..84795c148 100644--- a/packages/memory-storage/CHANGELOG.md+++ b/packages/memory-storage/CHANGELOG.md@@ -3,6 +3,17 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Bug Fixes++* cleanup worker stuff from memory storage to fix `vitest` ([#2004](https://github.com/apify/crawlee/issues/2004)) ([d2e098c](https://github.com/apify/crawlee/commit/d2e098cab62c700a5c58fcf43a5bcf9f492d71ec)), closes [#1999](https://github.com/apify/crawlee/issues/1999)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/memory-storagediff --git a/packages/memory-storage/package.json b/packages/memory-storage/package.jsonindex 34ec7e87d..ff3d18b1b 100644--- a/packages/memory-storage/package.json+++ b/packages/memory-storage/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/memory-storage",- "version": "3.4.2",+ "version": "3.5.0","description": "A simple in-memory storage implementation of the Apify API","engines": {"node": ">= 16"@@ -49,7 +49,7 @@},"dependencies": {"@apify/log": "^2.4.0",- "@crawlee/types": "^3.4.2",+ "@crawlee/types": "^3.5.0","@sapphire/async-queue": "^1.5.0","@sapphire/shapeshift": "^3.0.0","content-type": "^1.0.4",diff --git a/packages/playwright-crawler/CHANGELOG.md b/packages/playwright-crawler/CHANGELOG.mdindex 016843e2c..352bdb99e 100644--- a/packages/playwright-crawler/CHANGELOG.md+++ b/packages/playwright-crawler/CHANGELOG.md@@ -3,6 +3,18 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/playwrightdiff --git a/packages/playwright-crawler/package.json b/packages/playwright-crawler/package.jsonindex 2c12bb263..d3c7f27f7 100644--- a/packages/playwright-crawler/package.json+++ b/packages/playwright-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/playwright",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -55,10 +55,10 @@"dependencies": {"@apify/datastructures": "^2.0.0","@apify/log": "^2.4.0",- "@crawlee/browser": "^3.4.2",- "@crawlee/browser-pool": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/browser": "^3.5.0",+ "@crawlee/browser-pool": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","cheerio": "^1.0.0-rc.12","idcac-playwright": "^0.1.2","jquery": "^3.6.0",diff --git a/packages/puppeteer-crawler/CHANGELOG.md b/packages/puppeteer-crawler/CHANGELOG.mdindex d8a978f08..808a4e135 100644--- a/packages/puppeteer-crawler/CHANGELOG.md+++ b/packages/puppeteer-crawler/CHANGELOG.md@@ -3,6 +3,18 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/puppeteerdiff --git a/packages/puppeteer-crawler/package.json b/packages/puppeteer-crawler/package.jsonindex 0a1c6ad1f..4d80c46f6 100644--- a/packages/puppeteer-crawler/package.json+++ b/packages/puppeteer-crawler/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/puppeteer",- "version": "3.4.2",+ "version": "3.5.0","description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=16.0.0"@@ -55,10 +55,10 @@"dependencies": {"@apify/datastructures": "^2.0.0","@apify/log": "^2.4.0",- "@crawlee/browser": "^3.4.2",- "@crawlee/browser-pool": "^3.4.2",- "@crawlee/types": "^3.4.2",- "@crawlee/utils": "^3.4.2",+ "@crawlee/browser": "^3.5.0",+ "@crawlee/browser-pool": "^3.5.0",+ "@crawlee/types": "^3.5.0",+ "@crawlee/utils": "^3.5.0","cheerio": "^1.0.0-rc.12","devtools-protocol": "*","idcac-playwright": "^0.1.2",diff --git a/packages/templates/CHANGELOG.md b/packages/templates/CHANGELOG.mdindex 1f27d461e..1cdf1432d 100644--- a/packages/templates/CHANGELOG.md+++ b/packages/templates/CHANGELOG.md@@ -3,6 +3,14 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)++**Note:** Version bump only for package @crawlee/templates+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/templatesdiff --git a/packages/templates/package.json b/packages/templates/package.jsonindex b8430993d..e0705f8ff 100644--- a/packages/templates/package.json+++ b/packages/templates/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/templates",- "version": "3.4.2",+ "version": "3.5.0","description": "Templates for the crawlee projects","engines": {"node": ">=16.0.0"diff --git a/packages/types/CHANGELOG.md b/packages/types/CHANGELOG.mdindex 2526afd6c..2c32d18af 100644--- a/packages/types/CHANGELOG.md+++ b/packages/types/CHANGELOG.md@@ -3,6 +3,17 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)**Note:** Version bump only for package @crawlee/typesdiff --git a/packages/types/package.json b/packages/types/package.jsonindex 5aae7c92c..440d9684d 100644--- a/packages/types/package.json+++ b/packages/types/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/types",- "version": "3.4.2",+ "version": "3.5.0","description": "Shared types for the crawlee projects","engines": {"node": ">=16.0.0"diff --git a/packages/utils/CHANGELOG.md b/packages/utils/CHANGELOG.mdindex 690b52a2f..4de71631c 100644--- a/packages/utils/CHANGELOG.md+++ b/packages/utils/CHANGELOG.md@@ -3,6 +3,17 @@All notable changes to this project will be documented in this file.See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)+++### Features++* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)+++++## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)diff --git a/packages/utils/package.json b/packages/utils/package.jsonindex b4df24a95..a92066677 100644--- a/packages/utils/package.json+++ b/packages/utils/package.json@@ -1,6 +1,6 @@{"name": "@crawlee/utils",- "version": "3.4.2",+ "version": "3.5.0","description": "A set of shared utilities that can be used by crawlers","engines": {"node": ">=16.0.0"@@ -49,7 +49,7 @@"dependencies": {"@apify/log": "^2.4.0","@apify/ps-tree": "^1.2.0",- "@crawlee/types": "^3.4.2",+ "@crawlee/types": "^3.5.0","cheerio": "^1.0.0-rc.12","got-scraping": "^3.2.9","ow": "^0.28.1",diff --git a/yarn.lock b/yarn.lockindex 4341141fb..b11195814 100644--- a/yarn.lock+++ b/yarn.lock@@ -738,16 +738,16 @@ __metadata:languageName: nodelinkType: hard-"@crawlee/basic@^3.4.2, @crawlee/basic@workspace:packages/basic-crawler":+"@crawlee/basic@^3.5.0, @crawlee/basic@workspace:packages/basic-crawler":version: 0.0.0-use.localresolution: "@crawlee/basic@workspace:packages/basic-crawler"dependencies:"@apify/log": ^2.4.0"@apify/timeout": ^0.3.0"@apify/utilities": ^2.7.10- "@crawlee/core": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/core": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0got-scraping: ^3.2.9ow: ^0.28.1tldts: ^6.0.0@@ -756,13 +756,13 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/browser-pool@^3.4.2, @crawlee/browser-pool@workspace:packages/browser-pool":+"@crawlee/browser-pool@^3.5.0, @crawlee/browser-pool@workspace:packages/browser-pool":version: 0.0.0-use.localresolution: "@crawlee/browser-pool@workspace:packages/browser-pool"dependencies:"@apify/log": ^2.4.0"@apify/timeout": ^0.3.0- "@crawlee/types": ^3.4.2+ "@crawlee/types": ^3.5.0fingerprint-generator: ^2.0.6fingerprint-injector: ^2.0.5lodash.merge: ^4.6.2@@ -784,37 +784,37 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/browser@^3.4.2, @crawlee/browser@workspace:packages/browser-crawler":+"@crawlee/browser@^3.5.0, @crawlee/browser@workspace:packages/browser-crawler":version: 0.0.0-use.localresolution: "@crawlee/browser@workspace:packages/browser-crawler"dependencies:"@apify/timeout": ^0.3.0- "@crawlee/basic": ^3.4.2- "@crawlee/browser-pool": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/basic": ^3.5.0+ "@crawlee/browser-pool": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0ow: ^0.28.1tslib: ^2.4.0languageName: unknownlinkType: soft-"@crawlee/cheerio@^3.4.2, @crawlee/cheerio@workspace:packages/cheerio-crawler":+"@crawlee/cheerio@^3.5.0, @crawlee/cheerio@workspace:packages/cheerio-crawler":version: 0.0.0-use.localresolution: "@crawlee/cheerio@workspace:packages/cheerio-crawler"dependencies:- "@crawlee/http": ^3.4.2- "@crawlee/types": ^3.4.2+ "@crawlee/http": ^3.5.0+ "@crawlee/types": ^3.5.0cheerio: ^1.0.0-rc.12htmlparser2: ^9.0.0tslib: ^2.4.0languageName: unknownlinkType: soft-"@crawlee/cli@^3.4.2, @crawlee/cli@workspace:packages/cli":+"@crawlee/cli@^3.5.0, @crawlee/cli@workspace:packages/cli":version: 0.0.0-use.localresolution: "@crawlee/cli@workspace:packages/cli"dependencies:- "@crawlee/templates": ^3.4.2+ "@crawlee/templates": ^3.5.0ansi-colors: ^4.1.3fs-extra: ^11.0.0inquirer: ^8.2.4@@ -826,7 +826,7 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/core@^3.4.2, @crawlee/core@workspace:packages/core":+"@crawlee/core@^3.4.2, @crawlee/core@^3.5.0, @crawlee/core@workspace:packages/core":version: 0.0.0-use.localresolution: "@crawlee/core@workspace:packages/core"dependencies:@@ -836,9 +836,9 @@ __metadata:"@apify/pseudo_url": ^2.0.30"@apify/timeout": ^0.3.0"@apify/utilities": ^2.7.10- "@crawlee/memory-storage": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/memory-storage": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0"@sapphire/async-queue": ^1.5.0"@types/tough-cookie": ^4.0.2"@vladfrangu/async_event_emitter": ^2.2.2@@ -856,15 +856,15 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/http@^3.4.2, @crawlee/http@workspace:packages/http-crawler":+"@crawlee/http@^3.5.0, @crawlee/http@workspace:packages/http-crawler":version: 0.0.0-use.localresolution: "@crawlee/http@workspace:packages/http-crawler"dependencies:"@apify/timeout": ^0.3.0"@apify/utilities": ^2.7.10- "@crawlee/basic": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/basic": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0"@types/content-type": ^1.1.5cheerio: ^1.0.0-rc.12content-type: ^1.0.4@@ -877,14 +877,14 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/jsdom@^3.4.2, @crawlee/jsdom@workspace:packages/jsdom-crawler":+"@crawlee/jsdom@^3.5.0, @crawlee/jsdom@workspace:packages/jsdom-crawler":version: 0.0.0-use.localresolution: "@crawlee/jsdom@workspace:packages/jsdom-crawler"dependencies:"@apify/timeout": ^0.3.0"@apify/utilities": ^2.7.10- "@crawlee/http": ^3.4.2- "@crawlee/types": ^3.4.2+ "@crawlee/http": ^3.5.0+ "@crawlee/types": ^3.5.0"@types/jsdom": ^21.0.0cheerio: ^1.0.0-rc.12jsdom: ^22.0.0@@ -893,26 +893,26 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/linkedom@^3.4.2, @crawlee/linkedom@workspace:packages/linkedom-crawler":+"@crawlee/linkedom@^3.5.0, @crawlee/linkedom@workspace:packages/linkedom-crawler":version: 0.0.0-use.localresolution: "@crawlee/linkedom@workspace:packages/linkedom-crawler"dependencies:"@apify/timeout": ^0.3.0"@apify/utilities": ^2.7.10- "@crawlee/http": ^3.4.2- "@crawlee/types": ^3.4.2+ "@crawlee/http": ^3.5.0+ "@crawlee/types": ^3.5.0linkedom: ^0.15.0ow: ^0.28.2tslib: ^2.4.0languageName: unknownlinkType: soft-"@crawlee/memory-storage@^3.4.2, @crawlee/memory-storage@workspace:packages/memory-storage":+"@crawlee/memory-storage@^3.5.0, @crawlee/memory-storage@workspace:packages/memory-storage":version: 0.0.0-use.localresolution: "@crawlee/memory-storage@workspace:packages/memory-storage"dependencies:"@apify/log": ^2.4.0- "@crawlee/types": ^3.4.2+ "@crawlee/types": ^3.5.0"@sapphire/async-queue": ^1.5.0"@sapphire/shapeshift": ^3.0.0content-type: ^1.0.4@@ -924,16 +924,16 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/playwright@^3.4.2, @crawlee/playwright@workspace:packages/playwright-crawler":+"@crawlee/playwright@^3.5.0, @crawlee/playwright@workspace:packages/playwright-crawler":version: 0.0.0-use.localresolution: "@crawlee/playwright@workspace:packages/playwright-crawler"dependencies:"@apify/datastructures": ^2.0.0"@apify/log": ^2.4.0- "@crawlee/browser": ^3.4.2- "@crawlee/browser-pool": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/browser": ^3.5.0+ "@crawlee/browser-pool": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0cheerio: ^1.0.0-rc.12idcac-playwright: ^0.1.2jquery: ^3.6.0@@ -947,16 +947,16 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/puppeteer@^3.4.2, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":+"@crawlee/puppeteer@^3.5.0, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":version: 0.0.0-use.localresolution: "@crawlee/puppeteer@workspace:packages/puppeteer-crawler"dependencies:"@apify/datastructures": ^2.0.0"@apify/log": ^2.4.0- "@crawlee/browser": ^3.4.2- "@crawlee/browser-pool": ^3.4.2- "@crawlee/types": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/browser": ^3.5.0+ "@crawlee/browser-pool": ^3.5.0+ "@crawlee/types": ^3.5.0+ "@crawlee/utils": ^3.5.0cheerio: ^1.0.0-rc.12devtools-protocol: "*"idcac-playwright: ^0.1.2@@ -1027,7 +1027,7 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/templates@^3.4.2, @crawlee/templates@workspace:packages/templates":+"@crawlee/templates@^3.5.0, @crawlee/templates@workspace:packages/templates":version: 0.0.0-use.localresolution: "@crawlee/templates@workspace:packages/templates"dependencies:@@ -1039,7 +1039,7 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/types@^3.3.0, @crawlee/types@^3.4.2, @crawlee/types@workspace:packages/types":+"@crawlee/types@^3.3.0, @crawlee/types@^3.4.2, @crawlee/types@^3.5.0, @crawlee/types@workspace:packages/types":version: 0.0.0-use.localresolution: "@crawlee/types@workspace:packages/types"dependencies:@@ -1047,13 +1047,13 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/utils@^3.4.2, @crawlee/utils@workspace:packages/utils":+"@crawlee/utils@^3.4.2, @crawlee/utils@^3.5.0, @crawlee/utils@workspace:packages/utils":version: 0.0.0-use.localresolution: "@crawlee/utils@workspace:packages/utils"dependencies:"@apify/log": ^2.4.0"@apify/ps-tree": ^1.2.0- "@crawlee/types": ^3.4.2+ "@crawlee/types": ^3.5.0cheerio: ^1.0.0-rc.12got-scraping: ^3.2.9ow: ^0.28.1@@ -4298,17 +4298,17 @@ __metadata:version: 0.0.0-use.localresolution: "crawlee@workspace:packages/crawlee"dependencies:- "@crawlee/basic": ^3.4.2- "@crawlee/browser": ^3.4.2- "@crawlee/cheerio": ^3.4.2- "@crawlee/cli": ^3.4.2- "@crawlee/core": ^3.4.2- "@crawlee/http": ^3.4.2- "@crawlee/jsdom": ^3.4.2- "@crawlee/linkedom": ^3.4.2- "@crawlee/playwright": ^3.4.2- "@crawlee/puppeteer": ^3.4.2- "@crawlee/utils": ^3.4.2+ "@crawlee/basic": ^3.5.0+ "@crawlee/browser": ^3.5.0+ "@crawlee/cheerio": ^3.5.0+ "@crawlee/cli": ^3.5.0+ "@crawlee/core": ^3.5.0+ "@crawlee/http": ^3.5.0+ "@crawlee/jsdom": ^3.5.0+ "@crawlee/linkedom": ^3.5.0+ "@crawlee/playwright": ^3.5.0+ "@crawlee/puppeteer": ^3.5.0+ "@crawlee/utils": ^3.5.0import-local: ^3.1.0tslib: ^2.4.0peerDependencies: