Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore(release): v3.5.0

Apify Release Bot58cc3d4a74ee
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 395df35cb..1f06707e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,28 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Bug Fixes
+
+* cleanup worker stuff from memory storage to fix `vitest` ([#2004](https://github.com/apify/crawlee/issues/2004)) ([d2e098c](https://github.com/apify/crawlee/commit/d2e098cab62c700a5c58fcf43a5bcf9f492d71ec)), closes [#1999](https://github.com/apify/crawlee/issues/1999)
+* **core:** add requests from URL list (`requestsFromUrl`) to the queue in batches ([418fbf8](https://github.com/apify/crawlee/commit/418fbf89d8680f8c460e37cfbf3e521f45770eb2)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+* **core:** support relative links in `enqueueLinks` explicitly provided via `urls` option ([#2014](https://github.com/apify/crawlee/issues/2014)) ([cbd9d08](https://github.com/apify/crawlee/commit/cbd9d08065694b8c86e32c773875cecd41e5fcc9)), closes [#2005](https://github.com/apify/crawlee/issues/2005)
+
+
+### Features
+
+* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))
+* add support for `sameDomainDelay` ([#2003](https://github.com/apify/crawlee/issues/2003)) ([e796883](https://github.com/apify/crawlee/commit/e79688324790e5d07fc11192769cf051617e96e4)), closes [#1993](https://github.com/apify/crawlee/issues/1993)
+* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))
+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/lerna.json b/lerna.json
index be7d39a90..4e96cb30c 100644
--- a/lerna.json
+++ b/lerna.json
@@ -2,7 +2,7 @@
"packages": [
"packages/*"
],
- "version": "3.4.2",
+ "version": "3.5.0",
"command": {
"version": {
"conventionalCommits": true,
diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.md
index b8dda4ffa..25131bb75 100644
--- a/packages/basic-crawler/CHANGELOG.md
+++ b/packages/basic-crawler/CHANGELOG.md
@@ -3,6 +3,19 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* add support for `sameDomainDelay` ([#2003](https://github.com/apify/crawlee/issues/2003)) ([e796883](https://github.com/apify/crawlee/commit/e79688324790e5d07fc11192769cf051617e96e4)), closes [#1993](https://github.com/apify/crawlee/issues/1993)
+* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
index 210381354..d1ed6245f 100644
--- a/packages/basic-crawler/package.json
+++ b/packages/basic-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/basic",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -48,9 +48,9 @@
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/core": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/core": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"got-scraping": "^3.2.9",
"ow": "^0.28.1",
"tldts": "^6.0.0",
diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.md
index 6dd6db148..c26f71e75 100644
--- a/packages/browser-crawler/CHANGELOG.md
+++ b/packages/browser-crawler/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.json
index 1d30a3165..3d2b87d8e 100644
--- a/packages/browser-crawler/package.json
+++ b/packages/browser-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/browser",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -54,10 +54,10 @@
},
"dependencies": {
"@apify/timeout": "^0.3.0",
- "@crawlee/basic": "^3.4.2",
- "@crawlee/browser-pool": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/basic": "^3.5.0",
+ "@crawlee/browser-pool": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"ow": "^0.28.1",
"tslib": "^2.4.0"
}
diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.md
index 4b6cfb65d..5387728d9 100644
--- a/packages/browser-pool/CHANGELOG.md
+++ b/packages/browser-pool/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/browser-pool
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/browser-pool
diff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.json
index 79daed9bc..3a31bd05e 100644
--- a/packages/browser-pool/package.json
+++ b/packages/browser-pool/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/browser-pool",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -38,7 +38,7 @@
"dependencies": {
"@apify/log": "^2.4.0",
"@apify/timeout": "^0.3.0",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/types": "^3.5.0",
"fingerprint-generator": "^2.0.6",
"fingerprint-injector": "^2.0.5",
"lodash.merge": "^4.6.2",
diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.md
index a9bd95cb1..d3fbc0299 100644
--- a/packages/cheerio-crawler/CHANGELOG.md
+++ b/packages/cheerio-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/cheerio
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/cheerio
diff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.json
index 75f3a2e20..871a90bf6 100644
--- a/packages/cheerio-crawler/package.json
+++ b/packages/cheerio-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/cheerio",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -53,8 +53,8 @@
"access": "public"
},
"dependencies": {
- "@crawlee/http": "^3.4.2",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/http": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
"cheerio": "^1.0.0-rc.12",
"htmlparser2": "^9.0.0",
"tslib": "^2.4.0"
diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md
index 74facbb9a..c8d21ee2a 100644
--- a/packages/cli/CHANGELOG.md
+++ b/packages/cli/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/cli
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/cli
diff --git a/packages/cli/package.json b/packages/cli/package.json
index d91b5986d..80646b56f 100644
--- a/packages/cli/package.json
+++ b/packages/cli/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/cli",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -51,7 +51,7 @@
"access": "public"
},
"dependencies": {
- "@crawlee/templates": "^3.4.2",
+ "@crawlee/templates": "^3.5.0",
"ansi-colors": "^4.1.3",
"fs-extra": "^11.0.0",
"inquirer": "^8.2.4",
diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.md
index 2303ded86..9abb93eef 100644
--- a/packages/core/CHANGELOG.md
+++ b/packages/core/CHANGELOG.md
@@ -3,6 +3,24 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Bug Fixes
+
+* **core:** add requests from URL list (`requestsFromUrl`) to the queue in batches ([418fbf8](https://github.com/apify/crawlee/commit/418fbf89d8680f8c460e37cfbf3e521f45770eb2)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+* **core:** support relative links in `enqueueLinks` explicitly provided via `urls` option ([#2014](https://github.com/apify/crawlee/issues/2014)) ([cbd9d08](https://github.com/apify/crawlee/commit/cbd9d08065694b8c86e32c773875cecd41e5fcc9)), closes [#2005](https://github.com/apify/crawlee/issues/2005)
+
+
+### Features
+
+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/packages/core/package.json b/packages/core/package.json
index 33d42133d..026fe4722 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/core",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -59,9 +59,9 @@
"@apify/pseudo_url": "^2.0.30",
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/memory-storage": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/memory-storage": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"@sapphire/async-queue": "^1.5.0",
"@types/tough-cookie": "^4.0.2",
"@vladfrangu/async_event_emitter": "^2.2.2",
diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.md
index 39cb8f60e..f8a8d46cf 100644
--- a/packages/crawlee/CHANGELOG.md
+++ b/packages/crawlee/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package crawlee
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package crawlee
diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
index 5bb31a289..9b222aa86 100644
--- a/packages/crawlee/package.json
+++ b/packages/crawlee/package.json
@@ -1,6 +1,6 @@
{
"name": "crawlee",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -54,17 +54,17 @@
"access": "public"
},
"dependencies": {
- "@crawlee/basic": "^3.4.2",
- "@crawlee/browser": "^3.4.2",
- "@crawlee/cheerio": "^3.4.2",
- "@crawlee/cli": "^3.4.2",
- "@crawlee/core": "^3.4.2",
- "@crawlee/http": "^3.4.2",
- "@crawlee/jsdom": "^3.4.2",
- "@crawlee/linkedom": "^3.4.2",
- "@crawlee/playwright": "^3.4.2",
- "@crawlee/puppeteer": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/basic": "^3.5.0",
+ "@crawlee/browser": "^3.5.0",
+ "@crawlee/cheerio": "^3.5.0",
+ "@crawlee/cli": "^3.5.0",
+ "@crawlee/core": "^3.5.0",
+ "@crawlee/http": "^3.5.0",
+ "@crawlee/jsdom": "^3.5.0",
+ "@crawlee/linkedom": "^3.5.0",
+ "@crawlee/playwright": "^3.5.0",
+ "@crawlee/puppeteer": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"import-local": "^3.1.0",
"tslib": "^2.4.0"
},
diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.md
index c61d8c054..616e8c18c 100644
--- a/packages/http-crawler/CHANGELOG.md
+++ b/packages/http-crawler/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.json
index c026d02fb..4f6eab7be 100644
--- a/packages/http-crawler/package.json
+++ b/packages/http-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/http",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,9 +55,9 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/basic": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/basic": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"@types/content-type": "^1.1.5",
"cheerio": "^1.0.0-rc.12",
"content-type": "^1.0.4",
diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.md
index fc94018bf..9c4382f63 100644
--- a/packages/jsdom-crawler/CHANGELOG.md
+++ b/packages/jsdom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/jsdom
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/jsdom
diff --git a/packages/jsdom-crawler/package.json b/packages/jsdom-crawler/package.json
index 335ee4e09..f782505a6 100644
--- a/packages/jsdom-crawler/package.json
+++ b/packages/jsdom-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/jsdom",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,8 +55,8 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/http": "^3.4.2",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/http": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
"@types/jsdom": "^21.0.0",
"cheerio": "^1.0.0-rc.12",
"jsdom": "^22.0.0",
diff --git a/packages/linkedom-crawler/CHANGELOG.md b/packages/linkedom-crawler/CHANGELOG.md
index 6184c64c9..423809bad 100644
--- a/packages/linkedom-crawler/CHANGELOG.md
+++ b/packages/linkedom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/linkedom
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/linkedom
diff --git a/packages/linkedom-crawler/package.json b/packages/linkedom-crawler/package.json
index 3864a79d5..5bd921d23 100644
--- a/packages/linkedom-crawler/package.json
+++ b/packages/linkedom-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/linkedom",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,8 +55,8 @@
"dependencies": {
"@apify/timeout": "^0.3.0",
"@apify/utilities": "^2.7.10",
- "@crawlee/http": "^3.4.2",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/http": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
"linkedom": "^0.15.0",
"ow": "^0.28.2",
"tslib": "^2.4.0"
diff --git a/packages/memory-storage/CHANGELOG.md b/packages/memory-storage/CHANGELOG.md
index 30462526b..84795c148 100644
--- a/packages/memory-storage/CHANGELOG.md
+++ b/packages/memory-storage/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Bug Fixes
+
+* cleanup worker stuff from memory storage to fix `vitest` ([#2004](https://github.com/apify/crawlee/issues/2004)) ([d2e098c](https://github.com/apify/crawlee/commit/d2e098cab62c700a5c58fcf43a5bcf9f492d71ec)), closes [#1999](https://github.com/apify/crawlee/issues/1999)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/memory-storage
diff --git a/packages/memory-storage/package.json b/packages/memory-storage/package.json
index 34ec7e87d..ff3d18b1b 100644
--- a/packages/memory-storage/package.json
+++ b/packages/memory-storage/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/memory-storage",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "A simple in-memory storage implementation of the Apify API",
"engines": {
"node": ">= 16"
@@ -49,7 +49,7 @@
},
"dependencies": {
"@apify/log": "^2.4.0",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/types": "^3.5.0",
"@sapphire/async-queue": "^1.5.0",
"@sapphire/shapeshift": "^3.0.0",
"content-type": "^1.0.4",
diff --git a/packages/playwright-crawler/CHANGELOG.md b/packages/playwright-crawler/CHANGELOG.md
index 016843e2c..352bdb99e 100644
--- a/packages/playwright-crawler/CHANGELOG.md
+++ b/packages/playwright-crawler/CHANGELOG.md
@@ -3,6 +3,18 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))
+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/playwright
diff --git a/packages/playwright-crawler/package.json b/packages/playwright-crawler/package.json
index 2c12bb263..d3c7f27f7 100644
--- a/packages/playwright-crawler/package.json
+++ b/packages/playwright-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/playwright",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,10 +55,10 @@
"dependencies": {
"@apify/datastructures": "^2.0.0",
"@apify/log": "^2.4.0",
- "@crawlee/browser": "^3.4.2",
- "@crawlee/browser-pool": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/browser": "^3.5.0",
+ "@crawlee/browser-pool": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"cheerio": "^1.0.0-rc.12",
"idcac-playwright": "^0.1.2",
"jquery": "^3.6.0",
diff --git a/packages/puppeteer-crawler/CHANGELOG.md b/packages/puppeteer-crawler/CHANGELOG.md
index d8a978f08..808a4e135 100644
--- a/packages/puppeteer-crawler/CHANGELOG.md
+++ b/packages/puppeteer-crawler/CHANGELOG.md
@@ -3,6 +3,18 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* add `closeCookieModals` context helper for Playwright and Puppeteer ([#1927](https://github.com/apify/crawlee/issues/1927)) ([98d93bb](https://github.com/apify/crawlee/commit/98d93bb6713ec219baa83db2ad2cd1d7621a3339))
+* **core:** use `RequestQueue.addBatchedRequests()` in `enqueueLinks` helper ([4d61ca9](https://github.com/apify/crawlee/commit/4d61ca934072f8bbb680c842d8b1c9a4452ee73a)), closes [#1995](https://github.com/apify/crawlee/issues/1995)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/puppeteer
diff --git a/packages/puppeteer-crawler/package.json b/packages/puppeteer-crawler/package.json
index 0a1c6ad1f..4d80c46f6 100644
--- a/packages/puppeteer-crawler/package.json
+++ b/packages/puppeteer-crawler/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/puppeteer",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=16.0.0"
@@ -55,10 +55,10 @@
"dependencies": {
"@apify/datastructures": "^2.0.0",
"@apify/log": "^2.4.0",
- "@crawlee/browser": "^3.4.2",
- "@crawlee/browser-pool": "^3.4.2",
- "@crawlee/types": "^3.4.2",
- "@crawlee/utils": "^3.4.2",
+ "@crawlee/browser": "^3.5.0",
+ "@crawlee/browser-pool": "^3.5.0",
+ "@crawlee/types": "^3.5.0",
+ "@crawlee/utils": "^3.5.0",
"cheerio": "^1.0.0-rc.12",
"devtools-protocol": "*",
"idcac-playwright": "^0.1.2",
diff --git a/packages/templates/CHANGELOG.md b/packages/templates/CHANGELOG.md
index 1f27d461e..1cdf1432d 100644
--- a/packages/templates/CHANGELOG.md
+++ b/packages/templates/CHANGELOG.md
@@ -3,6 +3,14 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+**Note:** Version bump only for package @crawlee/templates
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/templates
diff --git a/packages/templates/package.json b/packages/templates/package.json
index b8430993d..e0705f8ff 100644
--- a/packages/templates/package.json
+++ b/packages/templates/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/templates",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "Templates for the crawlee projects",
"engines": {
"node": ">=16.0.0"
diff --git a/packages/types/CHANGELOG.md b/packages/types/CHANGELOG.md
index 2526afd6c..2c32d18af 100644
--- a/packages/types/CHANGELOG.md
+++ b/packages/types/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* **basic-crawler:** allow configuring the automatic status message ([#2001](https://github.com/apify/crawlee/issues/2001)) ([3eb4e4c](https://github.com/apify/crawlee/commit/3eb4e4c558b4bc0673fbff75b1db19c46004a1da))
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
**Note:** Version bump only for package @crawlee/types
diff --git a/packages/types/package.json b/packages/types/package.json
index 5aae7c92c..440d9684d 100644
--- a/packages/types/package.json
+++ b/packages/types/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/types",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "Shared types for the crawlee projects",
"engines": {
"node": ">=16.0.0"
diff --git a/packages/utils/CHANGELOG.md b/packages/utils/CHANGELOG.md
index 690b52a2f..4de71631c 100644
--- a/packages/utils/CHANGELOG.md
+++ b/packages/utils/CHANGELOG.md
@@ -3,6 +3,17 @@
All notable changes to this project will be documented in this file.
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+# [3.5.0](https://github.com/apify/crawlee/compare/v3.4.2...v3.5.0) (2023-07-31)
+
+
+### Features
+
+* retire session on proxy error ([#2002](https://github.com/apify/crawlee/issues/2002)) ([8c0928b](https://github.com/apify/crawlee/commit/8c0928b24ceabefc454f8114ac30a27023709010)), closes [#1912](https://github.com/apify/crawlee/issues/1912)
+
+
+
+
+
## [3.4.2](https://github.com/apify/crawlee/compare/v3.4.1...v3.4.2) (2023-07-19)
diff --git a/packages/utils/package.json b/packages/utils/package.json
index b4df24a95..a92066677 100644
--- a/packages/utils/package.json
+++ b/packages/utils/package.json
@@ -1,6 +1,6 @@
{
"name": "@crawlee/utils",
- "version": "3.4.2",
+ "version": "3.5.0",
"description": "A set of shared utilities that can be used by crawlers",
"engines": {
"node": ">=16.0.0"
@@ -49,7 +49,7 @@
"dependencies": {
"@apify/log": "^2.4.0",
"@apify/ps-tree": "^1.2.0",
- "@crawlee/types": "^3.4.2",
+ "@crawlee/types": "^3.5.0",
"cheerio": "^1.0.0-rc.12",
"got-scraping": "^3.2.9",
"ow": "^0.28.1",
diff --git a/yarn.lock b/yarn.lock
index 4341141fb..b11195814 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -738,16 +738,16 @@ __metadata:
languageName: node
linkType: hard
-"@crawlee/basic@^3.4.2, @crawlee/basic@workspace:packages/basic-crawler":
+"@crawlee/basic@^3.5.0, @crawlee/basic@workspace:packages/basic-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/basic@workspace:packages/basic-crawler"
dependencies:
"@apify/log": ^2.4.0
"@apify/timeout": ^0.3.0
"@apify/utilities": ^2.7.10
- "@crawlee/core": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/core": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
got-scraping: ^3.2.9
ow: ^0.28.1
tldts: ^6.0.0
@@ -756,13 +756,13 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/browser-pool@^3.4.2, @crawlee/browser-pool@workspace:packages/browser-pool":
+"@crawlee/browser-pool@^3.5.0, @crawlee/browser-pool@workspace:packages/browser-pool":
version: 0.0.0-use.local
resolution: "@crawlee/browser-pool@workspace:packages/browser-pool"
dependencies:
"@apify/log": ^2.4.0
"@apify/timeout": ^0.3.0
- "@crawlee/types": ^3.4.2
+ "@crawlee/types": ^3.5.0
fingerprint-generator: ^2.0.6
fingerprint-injector: ^2.0.5
lodash.merge: ^4.6.2
@@ -784,37 +784,37 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/browser@^3.4.2, @crawlee/browser@workspace:packages/browser-crawler":
+"@crawlee/browser@^3.5.0, @crawlee/browser@workspace:packages/browser-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/browser@workspace:packages/browser-crawler"
dependencies:
"@apify/timeout": ^0.3.0
- "@crawlee/basic": ^3.4.2
- "@crawlee/browser-pool": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/basic": ^3.5.0
+ "@crawlee/browser-pool": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
ow: ^0.28.1
tslib: ^2.4.0
languageName: unknown
linkType: soft
-"@crawlee/cheerio@^3.4.2, @crawlee/cheerio@workspace:packages/cheerio-crawler":
+"@crawlee/cheerio@^3.5.0, @crawlee/cheerio@workspace:packages/cheerio-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/cheerio@workspace:packages/cheerio-crawler"
dependencies:
- "@crawlee/http": ^3.4.2
- "@crawlee/types": ^3.4.2
+ "@crawlee/http": ^3.5.0
+ "@crawlee/types": ^3.5.0
cheerio: ^1.0.0-rc.12
htmlparser2: ^9.0.0
tslib: ^2.4.0
languageName: unknown
linkType: soft
-"@crawlee/cli@^3.4.2, @crawlee/cli@workspace:packages/cli":
+"@crawlee/cli@^3.5.0, @crawlee/cli@workspace:packages/cli":
version: 0.0.0-use.local
resolution: "@crawlee/cli@workspace:packages/cli"
dependencies:
- "@crawlee/templates": ^3.4.2
+ "@crawlee/templates": ^3.5.0
ansi-colors: ^4.1.3
fs-extra: ^11.0.0
inquirer: ^8.2.4
@@ -826,7 +826,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/core@^3.4.2, @crawlee/core@workspace:packages/core":
+"@crawlee/core@^3.4.2, @crawlee/core@^3.5.0, @crawlee/core@workspace:packages/core":
version: 0.0.0-use.local
resolution: "@crawlee/core@workspace:packages/core"
dependencies:
@@ -836,9 +836,9 @@ __metadata:
"@apify/pseudo_url": ^2.0.30
"@apify/timeout": ^0.3.0
"@apify/utilities": ^2.7.10
- "@crawlee/memory-storage": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/memory-storage": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
"@sapphire/async-queue": ^1.5.0
"@types/tough-cookie": ^4.0.2
"@vladfrangu/async_event_emitter": ^2.2.2
@@ -856,15 +856,15 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/http@^3.4.2, @crawlee/http@workspace:packages/http-crawler":
+"@crawlee/http@^3.5.0, @crawlee/http@workspace:packages/http-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/http@workspace:packages/http-crawler"
dependencies:
"@apify/timeout": ^0.3.0
"@apify/utilities": ^2.7.10
- "@crawlee/basic": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/basic": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
"@types/content-type": ^1.1.5
cheerio: ^1.0.0-rc.12
content-type: ^1.0.4
@@ -877,14 +877,14 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/jsdom@^3.4.2, @crawlee/jsdom@workspace:packages/jsdom-crawler":
+"@crawlee/jsdom@^3.5.0, @crawlee/jsdom@workspace:packages/jsdom-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/jsdom@workspace:packages/jsdom-crawler"
dependencies:
"@apify/timeout": ^0.3.0
"@apify/utilities": ^2.7.10
- "@crawlee/http": ^3.4.2
- "@crawlee/types": ^3.4.2
+ "@crawlee/http": ^3.5.0
+ "@crawlee/types": ^3.5.0
"@types/jsdom": ^21.0.0
cheerio: ^1.0.0-rc.12
jsdom: ^22.0.0
@@ -893,26 +893,26 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/linkedom@^3.4.2, @crawlee/linkedom@workspace:packages/linkedom-crawler":
+"@crawlee/linkedom@^3.5.0, @crawlee/linkedom@workspace:packages/linkedom-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/linkedom@workspace:packages/linkedom-crawler"
dependencies:
"@apify/timeout": ^0.3.0
"@apify/utilities": ^2.7.10
- "@crawlee/http": ^3.4.2
- "@crawlee/types": ^3.4.2
+ "@crawlee/http": ^3.5.0
+ "@crawlee/types": ^3.5.0
linkedom: ^0.15.0
ow: ^0.28.2
tslib: ^2.4.0
languageName: unknown
linkType: soft
-"@crawlee/memory-storage@^3.4.2, @crawlee/memory-storage@workspace:packages/memory-storage":
+"@crawlee/memory-storage@^3.5.0, @crawlee/memory-storage@workspace:packages/memory-storage":
version: 0.0.0-use.local
resolution: "@crawlee/memory-storage@workspace:packages/memory-storage"
dependencies:
"@apify/log": ^2.4.0
- "@crawlee/types": ^3.4.2
+ "@crawlee/types": ^3.5.0
"@sapphire/async-queue": ^1.5.0
"@sapphire/shapeshift": ^3.0.0
content-type: ^1.0.4
@@ -924,16 +924,16 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/playwright@^3.4.2, @crawlee/playwright@workspace:packages/playwright-crawler":
+"@crawlee/playwright@^3.5.0, @crawlee/playwright@workspace:packages/playwright-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/playwright@workspace:packages/playwright-crawler"
dependencies:
"@apify/datastructures": ^2.0.0
"@apify/log": ^2.4.0
- "@crawlee/browser": ^3.4.2
- "@crawlee/browser-pool": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/browser": ^3.5.0
+ "@crawlee/browser-pool": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
cheerio: ^1.0.0-rc.12
idcac-playwright: ^0.1.2
jquery: ^3.6.0
@@ -947,16 +947,16 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/puppeteer@^3.4.2, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":
+"@crawlee/puppeteer@^3.5.0, @crawlee/puppeteer@workspace:packages/puppeteer-crawler":
version: 0.0.0-use.local
resolution: "@crawlee/puppeteer@workspace:packages/puppeteer-crawler"
dependencies:
"@apify/datastructures": ^2.0.0
"@apify/log": ^2.4.0
- "@crawlee/browser": ^3.4.2
- "@crawlee/browser-pool": ^3.4.2
- "@crawlee/types": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/browser": ^3.5.0
+ "@crawlee/browser-pool": ^3.5.0
+ "@crawlee/types": ^3.5.0
+ "@crawlee/utils": ^3.5.0
cheerio: ^1.0.0-rc.12
devtools-protocol: "*"
idcac-playwright: ^0.1.2
@@ -1027,7 +1027,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/templates@^3.4.2, @crawlee/templates@workspace:packages/templates":
+"@crawlee/templates@^3.5.0, @crawlee/templates@workspace:packages/templates":
version: 0.0.0-use.local
resolution: "@crawlee/templates@workspace:packages/templates"
dependencies:
@@ -1039,7 +1039,7 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/types@^3.3.0, @crawlee/types@^3.4.2, @crawlee/types@workspace:packages/types":
+"@crawlee/types@^3.3.0, @crawlee/types@^3.4.2, @crawlee/types@^3.5.0, @crawlee/types@workspace:packages/types":
version: 0.0.0-use.local
resolution: "@crawlee/types@workspace:packages/types"
dependencies:
@@ -1047,13 +1047,13 @@ __metadata:
languageName: unknown
linkType: soft
-"@crawlee/utils@^3.4.2, @crawlee/utils@workspace:packages/utils":
+"@crawlee/utils@^3.4.2, @crawlee/utils@^3.5.0, @crawlee/utils@workspace:packages/utils":
version: 0.0.0-use.local
resolution: "@crawlee/utils@workspace:packages/utils"
dependencies:
"@apify/log": ^2.4.0
"@apify/ps-tree": ^1.2.0
- "@crawlee/types": ^3.4.2
+ "@crawlee/types": ^3.5.0
cheerio: ^1.0.0-rc.12
got-scraping: ^3.2.9
ow: ^0.28.1
@@ -4298,17 +4298,17 @@ __metadata:
version: 0.0.0-use.local
resolution: "crawlee@workspace:packages/crawlee"
dependencies:
- "@crawlee/basic": ^3.4.2
- "@crawlee/browser": ^3.4.2
- "@crawlee/cheerio": ^3.4.2
- "@crawlee/cli": ^3.4.2
- "@crawlee/core": ^3.4.2
- "@crawlee/http": ^3.4.2
- "@crawlee/jsdom": ^3.4.2
- "@crawlee/linkedom": ^3.4.2
- "@crawlee/playwright": ^3.4.2
- "@crawlee/puppeteer": ^3.4.2
- "@crawlee/utils": ^3.4.2
+ "@crawlee/basic": ^3.5.0
+ "@crawlee/browser": ^3.5.0
+ "@crawlee/cheerio": ^3.5.0
+ "@crawlee/cli": ^3.5.0
+ "@crawlee/core": ^3.5.0
+ "@crawlee/http": ^3.5.0
+ "@crawlee/jsdom": ^3.5.0
+ "@crawlee/linkedom": ^3.5.0
+ "@crawlee/playwright": ^3.5.0
+ "@crawlee/puppeteer": ^3.5.0
+ "@crawlee/utils": ^3.5.0
import-local: ^3.1.0
tslib: ^2.4.0
peerDependencies: