Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore: rename DOMCrawler to JSDOMCrawler (#1458)

Szymon Marczakb70740d5fafd
diff --git a/docs/examples/dom_crawler.mdx b/docs/examples/dom_crawler.mdx
deleted file mode 100644
index 9bda5246d..000000000
--- a/docs/examples/dom_crawler.mdx
+++ /dev/null
@@ -1,14 +0,0 @@
----
-id: dom-crawler
-title: DOM crawler
----
-
-import CodeBlock from '@theme/CodeBlock';
-import ApiLink from '@site/src/components/ApiLink';
-import DOMCrawlerSource from '!!raw-loader!./dom_crawler.ts';
-
-This example demonstrates how to use <ApiLink to="dom-crawler/class/DOMCrawler">`DOMCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, parse the HTML using the [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation and extract some data from it: the page title and all `h1` tags.
-
-<CodeBlock className="language-js">
- {DOMCrawlerSource}
-</CodeBlock>
diff --git a/docs/examples/jsdom_crawler.mdx b/docs/examples/jsdom_crawler.mdx
new file mode 100644
index 000000000..72decf749
--- /dev/null
+++ b/docs/examples/jsdom_crawler.mdx
@@ -0,0 +1,14 @@
+---
+id: jsdom-crawler
+title: JSDOM crawler
+---
+
+import CodeBlock from '@theme/CodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import JSDOMCrawlerSource from '!!raw-loader!./jsdom_crawler.ts';
+
+This example demonstrates how to use <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, parse the HTML using the [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation and extract some data from it: the page title and all `h1` tags.
+
+<CodeBlock className="language-js">
+ {JSDOMCrawlerSource}
+</CodeBlock>
diff --git a/docs/examples/dom_crawler.ts b/docs/examples/jsdom_crawler.ts
similarity index 95%
rename from docs/examples/dom_crawler.ts
rename to docs/examples/jsdom_crawler.ts
index d54102fcd..13b22fc74 100644
--- a/docs/examples/dom_crawler.ts
+++ b/docs/examples/jsdom_crawler.ts
@@ -1,4 +1,4 @@
-import { Dataset, DOMCrawler, log, LogLevel } from 'crawlee';
+import { Dataset, JSDOMCrawler, log, LogLevel } from 'crawlee';
// Crawlers come with various utilities, e.g. for logging.
// Here we use debug level of logging to improve the debugging experience.
@@ -7,7 +7,7 @@ log.setLevel(LogLevel.DEBUG);
// Create an instance of the CheerioCrawler class - a crawler
// that automatically loads the URLs and parses their HTML using the cheerio library.
-const crawler = new DOMCrawler({
+const crawler = new JSDOMCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
// automatically managed based on the available system memory and CPU (see AutoscaledPool class).
// Here we define some hard limits for the concurrency.
diff --git a/docs/examples/tsconfig.json b/docs/examples/tsconfig.json
index 7758dc0ba..e1b8e5b1a 100644
--- a/docs/examples/tsconfig.json
+++ b/docs/examples/tsconfig.json
@@ -7,7 +7,7 @@
"@crawlee/basic": ["packages/basic-crawler/src"],
"@crawlee/browser": ["packages/browser-crawler/src"],
"@crawlee/http": ["packages/http-crawler/src"],
- "@crawlee/dom": ["packages/dom-crawler/src"],
+ "@crawlee/jsdom": ["packages/jsdom-crawler/src"],
"@crawlee/cheerio": ["packages/cheerio-crawler/src"],
"@crawlee/playwright": ["packages/playwright-crawler/src"],
"@crawlee/puppeteer": ["packages/puppeteer-crawler/src"],
diff --git a/docs/guides/dom_crawler.mdx b/docs/guides/jsdom_crawler.mdx
similarity index 61%
rename from docs/guides/dom_crawler.mdx
rename to docs/guides/jsdom_crawler.mdx
index 618e26ffb..86c402d6d 100644
--- a/docs/guides/dom_crawler.mdx
+++ b/docs/guides/jsdom_crawler.mdx
@@ -1,17 +1,17 @@
---
-id: dom-crawler-guide
-title: "DOMCrawler guide"
-sidebar_label: "DOMCrawler"
+id: jsdom-crawler-guide
+title: "JSDOMCrawler guide"
+sidebar_label: "JSDOMCrawler"
description: Your first steps into the world of scraping with Crawlee
---
import ApiLink from '@site/src/components/ApiLink';
-&#8203;<ApiLink to="dom-crawler/class/DOMCrawler">`DOMCrawler`</ApiLink> is very useful for scraping with the Window API.
+&#8203;<ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> is very useful for scraping with the Window API.
## How the crawler works
-&#8203;<ApiLink to="dom-crawler/class/DOMCrawler">`DOMCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client. The URLs are fed to the crawler using <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the <ApiLink to="dom-crawler/interface/DOMCrawlerOptions#additionalMimeTypes">`additionalMimeTypes`</ApiLink> option.
+&#8203;<ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client. The URLs are fed to the crawler using <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the <ApiLink to="jsdom-crawler/interface/JSDOMCrawlerOptions#additionalMimeTypes">`additionalMimeTypes`</ApiLink> option.
:::info
@@ -29,9 +29,9 @@ document.title; // browsers
window.document.title; // JSDOM
```
-## When to use `DOMCrawler`
+## When to use `JSDOMCrawler`
-`DOMCrawler` really shines when `CheerioCrawler` is just not enough. There is an entire set of [APIs](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API) available!
+`JSDOMCrawler` really shines when `CheerioCrawler` is just not enough. There is an entire set of [APIs](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API) available!
**Advantages:**
@@ -58,4 +58,4 @@ Array.from(document.querySelectorAll('a[href]')).map((a) => a.href);
### Other examples
-Visit the [Examples](../examples) section to browse examples of `DOMCrawler` usage. Almost all examples show `DOMCrawler` code in their code tabs.
+Visit the [Examples](../examples) section to browse examples of `JSDOMCrawler` usage. Almost all examples show `JSDOMCrawler` code in their code tabs.
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdx
index 2c1d7ab8a..8b906ad12 100644
--- a/docs/guides/proxy_management.mdx
+++ b/docs/guides/proxy_management.mdx
@@ -11,18 +11,18 @@ import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import HttpSource from '!!raw-loader!./proxy_management_integration_http.ts';
-import DOMSource from '!!raw-loader!./proxy_management_integration_dom.ts';
+import JSDOMSource from '!!raw-loader!./proxy_management_integration_jsdom.ts';
import CheerioSource from '!!raw-loader!./proxy_management_integration_cheerio.ts';
import PlaywrightSource from '!!raw-loader!./proxy_management_integration_playwright.ts';
import PuppeteerSource from '!!raw-loader!./proxy_management_integration_puppeteer.ts';
import SessionStandaloneSource from '!!raw-loader!./proxy_management_session_standalone.ts';
import SessionHttpSource from '!!raw-loader!./proxy_management_session_http.ts';
-import SessionDOMSource from '!!raw-loader!./proxy_management_session_dom.ts';
+import SessionJSDOMSource from '!!raw-loader!./proxy_management_session_jsdom.ts';
import SessionCheerioSource from '!!raw-loader!./proxy_management_session_cheerio.ts';
import SessionPlaywrightSource from '!!raw-loader!./proxy_management_session_playwright.ts';
import SessionPuppeteerSource from '!!raw-loader!./proxy_management_session_puppeteer.ts';
import InspectionHttpSource from '!!raw-loader!./proxy_management_inspection_http.ts';
-import InspectionDOMSource from '!!raw-loader!./proxy_management_inspection_dom.ts';
+import InspectionJSDOMSource from '!!raw-loader!./proxy_management_inspection_jsdom.ts';
import InspectionCheerioSource from '!!raw-loader!./proxy_management_inspection_cheerio.ts';
import InspectionPlaywrightSource from '!!raw-loader!./proxy_management_inspection_playwright.ts';
import InspectionPuppeteerSource from '!!raw-loader!./proxy_management_inspection_puppeteer.ts';
@@ -61,7 +61,7 @@ All our proxy needs are managed by the <ApiLink to="core/class/ProxyConfiguratio
### Crawler integration
-`ProxyConfiguration` integrates seamlessly into <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="dom-crawler/class/DOMCrawler">`DOMCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
+`ProxyConfiguration` integrates seamlessly into <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
<Tabs groupId="proxy_session_management">
<TabItem value="http" label="HttpCrawler">
@@ -74,9 +74,9 @@ All our proxy needs are managed by the <ApiLink to="core/class/ProxyConfiguratio
{CheerioSource}
</CodeBlock>
</TabItem>
- <TabItem value="dom" label="DOMCrawler">
+ <TabItem value="jsdom" label="JSDOMCrawler">
<CodeBlock language="js">
- {DOMSource}
+ {JSDOMSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
@@ -110,9 +110,9 @@ When no `sessionId` is provided, our proxy URLs are rotated round-robin.
{SessionCheerioSource}
</CodeBlock>
</TabItem>
- <TabItem value="dom" label="DOMCrawler">
+ <TabItem value="jsdom" label="JSDOMCrawler">
<CodeBlock language="js">
- {SessionDOMSource}
+ {SessionJSDOMSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
@@ -134,7 +134,7 @@ When no `sessionId` is provided, our proxy URLs are rotated round-robin.
## Inspecting current proxy in Crawlers
-`HttpCrawler`, `CheerioCrawler`, `DOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
+`HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
in their `requestHandler` using a <ApiLink to="core/interface/ProxyInfo">`proxyInfo`</ApiLink> object.
With the `proxyInfo` object, we can easily access the proxy URL.
@@ -149,9 +149,9 @@ With the `proxyInfo` object, we can easily access the proxy URL.
{InspectionCheerioSource}
</CodeBlock>
</TabItem>
- <TabItem value="dom" label="DOMCrawler">
+ <TabItem value="jsdom" label="JSDOMCrawler">
<CodeBlock language="js">
- {InspectionDOMSource}
+ {InspectionJSDOMSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
diff --git a/docs/guides/proxy_management_inspection_dom.ts b/docs/guides/proxy_management_inspection_jsdom.ts
similarity index 66%
rename from docs/guides/proxy_management_inspection_dom.ts
rename to docs/guides/proxy_management_inspection_jsdom.ts
index 88260eefb..db7607143 100644
--- a/docs/guides/proxy_management_inspection_dom.ts
+++ b/docs/guides/proxy_management_inspection_jsdom.ts
@@ -1,8 +1,8 @@
-import { DOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
-const crawler = new DOMCrawler({
+const crawler = new JSDOMCrawler({
proxyConfiguration,
async requestHandler({ proxyInfo }) {
console.log(proxyInfo);
diff --git a/docs/guides/proxy_management_integration_dom.ts b/docs/guides/proxy_management_integration_jsdom.ts
similarity index 65%
rename from docs/guides/proxy_management_integration_dom.ts
rename to docs/guides/proxy_management_integration_jsdom.ts
index f2e4d4203..4164ef8cf 100644
--- a/docs/guides/proxy_management_integration_dom.ts
+++ b/docs/guides/proxy_management_integration_jsdom.ts
@@ -1,4 +1,4 @@
-import { DOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
@@ -7,7 +7,7 @@ const proxyConfiguration = new ProxyConfiguration({
],
});
-const crawler = new DOMCrawler({
+const crawler = new JSDOMCrawler({
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_dom.ts b/docs/guides/proxy_management_session_jsdom.ts
similarity index 64%
rename from docs/guides/proxy_management_session_dom.ts
rename to docs/guides/proxy_management_session_jsdom.ts
index 5e792919e..2321f97de 100644
--- a/docs/guides/proxy_management_session_dom.ts
+++ b/docs/guides/proxy_management_session_jsdom.ts
@@ -1,8 +1,8 @@
-import { DOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
-const crawler = new DOMCrawler({
+const crawler = new JSDOMCrawler({
useSessionPool: true,
persistCookiesPerSession: true,
proxyConfiguration,
diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdx
index 9429be3d9..104363f21 100644
--- a/docs/guides/session_management.mdx
+++ b/docs/guides/session_management.mdx
@@ -13,7 +13,7 @@ import CodeBlock from '@theme/CodeBlock';
import BasicSource from '!!raw-loader!./session_management_basic.ts';
import HttpSource from '!!raw-loader!./session_management_http.ts';
import CheerioSource from '!!raw-loader!./session_management_cheerio.ts';
-import DOMSource from '!!raw-loader!./session_management_dom.ts';
+import JSDOMSource from '!!raw-loader!./session_management_jsdom.ts';
import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';
import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';
import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';
@@ -31,7 +31,7 @@ Now let's take a look at the examples of how to use Session pool:
- with <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>;
- with <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>;
- with <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>;
-- with <ApiLink to="dom-crawler/class/DOMCrawler">`DOMCrawler`</ApiLink>;
+- with <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink>;
- with <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>;
- with <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>;
- without a crawler (standalone usage to manage sessions manually).
@@ -52,9 +52,9 @@ Now let's take a look at the examples of how to use Session pool:
{CheerioSource}
</CodeBlock>
</TabItem>
- <TabItem value="dom" label="DOMCrawler">
+ <TabItem value="jsdom" label="JSDOMCrawler">
<CodeBlock language="js">
- {DOMSource}
+ {JSDOMSource}
</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
diff --git a/docs/guides/session_management_dom.ts b/docs/guides/session_management_jsdom.ts
similarity index 90%
rename from docs/guides/session_management_dom.ts
rename to docs/guides/session_management_jsdom.ts
index 1e754581b..6f57ca9ba 100644
--- a/docs/guides/session_management_dom.ts
+++ b/docs/guides/session_management_jsdom.ts
@@ -1,8 +1,8 @@
-import { DOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
-const crawler = new DOMCrawler({
+const crawler = new JSDOMCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
// Activates the Session pool (default is true).
diff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading_v3.md
index 3d671e4aa..657fa8e21 100644
--- a/docs/upgrading/upgrading_v3.md
+++ b/docs/upgrading/upgrading_v3.md
@@ -22,8 +22,8 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
- `@crawlee/basic`: exports `BasicCrawler`
-- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/dom` and `@crawlee/cheerio`)
-- `@crawlee/dom`: exports `DOMCrawler`
+- `@crawlee/http`: exports `HttpCrawler` (which is used for creating `@crawlee/jsdom` and `@crawlee/cheerio`)
+- `@crawlee/jsdom`: exports `JSDOMCrawler`
- `@crawlee/cheerio`: exports `CheerioCrawler`
- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)
- `@crawlee/playwright`: exports `PlaywrightCrawler`
diff --git a/jest.config.js b/jest.config.js
index ded5b9d50..338d58d96 100644
--- a/jest.config.js
+++ b/jest.config.js
@@ -16,7 +16,7 @@ module.exports = {
'^@crawlee/basic$': '<rootDir>/packages/basic-crawler/src',
'^@crawlee/browser$': '<rootDir>/packages/browser-crawler/src',
'^@crawlee/http$': '<rootDir>/packages/http-crawler/src',
- '^@crawlee/dom$': '<rootDir>/packages/dom-crawler/src',
+ '^@crawlee/jsdom$': '<rootDir>/packages/jsdom-crawler/src',
'^@crawlee/cheerio$': '<rootDir>/packages/cheerio-crawler/src',
'^@crawlee/playwright$': '<rootDir>/packages/playwright-crawler/src',
'^@crawlee/puppeteer$': '<rootDir>/packages/puppeteer-crawler/src',
diff --git a/package-lock.json b/package-lock.json
index cc64c1422..bcc99f79e 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1030,8 +1030,8 @@
"resolved": "packages/core",
"link": true
},
- "node_modules/@crawlee/dom": {
- "resolved": "packages/dom-crawler",
+ "node_modules/@crawlee/jsdom": {
+ "resolved": "packages/jsdom-crawler",
"link": true
},
"node_modules/@crawlee/http": {
@@ -16485,7 +16485,7 @@
"@crawlee/cheerio": "^3.0.3",
"@crawlee/cli": "^3.0.3",
"@crawlee/core": "^3.0.3",
- "@crawlee/dom": "^3.0.3",
+ "@crawlee/jsdom": "^3.0.3",
"@crawlee/http": "^3.0.3",
"@crawlee/playwright": "^3.0.3",
"@crawlee/puppeteer": "^3.0.3",
@@ -16511,8 +16511,8 @@
}
}
},
- "packages/dom-crawler": {
- "name": "@crawlee/dom",
+ "packages/jsdom-crawler": {
+ "name": "@crawlee/jsdom",
"version": "3.0.3",
"license": "Apache-2.0",
"dependencies": {
@@ -17798,8 +17798,8 @@
}
}
},
- "@crawlee/dom": {
- "version": "file:packages/dom-crawler",
+ "@crawlee/jsdom": {
+ "version": "file:packages/jsdom-crawler",
"requires": {
"@apify/utilities": "^2.1.4",
"@crawlee/http": "^3.0.3",
@@ -22124,7 +22124,7 @@
"@crawlee/cheerio": "^3.0.3",
"@crawlee/cli": "^3.0.3",
"@crawlee/core": "^3.0.3",
- "@crawlee/dom": "^3.0.3",
+ "@crawlee/jsdom": "^3.0.3",
"@crawlee/http": "^3.0.3",
"@crawlee/playwright": "^3.0.3",
"@crawlee/puppeteer": "^3.0.3",
diff --git a/packages/core/README.md b/packages/core/README.md
index 8db53af17..6e88d36f6 100644
--- a/packages/core/README.md
+++ b/packages/core/README.md
@@ -8,9 +8,9 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever
- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`
- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`
- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`
-- [`@crawlee/dom`](https://crawlee.dev/api/dom-crawler): exports `DOMCrawler`
+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`
- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`
-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/dom`](https://crawlee.dev/api/dom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))
- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
index 81dfe385b..674c48412 100644
--- a/packages/crawlee/package.json
+++ b/packages/crawlee/package.json
@@ -62,7 +62,7 @@
"@crawlee/browser": "^3.0.3",
"@crawlee/http": "^3.0.3",
"@crawlee/cheerio": "^3.0.3",
- "@crawlee/dom": "^3.0.3",
+ "@crawlee/jsdom": "^3.0.3",
"@crawlee/playwright": "^3.0.3",
"@crawlee/puppeteer": "^3.0.3",
"@crawlee/utils": "^3.0.3",
diff --git a/packages/crawlee/src/index.ts b/packages/crawlee/src/index.ts
index 4c0c05dfe..e31d10a8b 100644
--- a/packages/crawlee/src/index.ts
+++ b/packages/crawlee/src/index.ts
@@ -10,7 +10,7 @@ export * from '@crawlee/utils';
export * from '@crawlee/basic';
export * from '@crawlee/browser';
export * from '@crawlee/http';
-export * from '@crawlee/dom';
+export * from '@crawlee/jsdom';
export * from '@crawlee/cheerio';
export * from '@crawlee/puppeteer';
export * from '@crawlee/playwright';
diff --git a/packages/dom-crawler/README.md b/packages/dom-crawler/README.md
deleted file mode 100644
index 3d4cf1d81..000000000
--- a/packages/dom-crawler/README.md
+++ /dev/null
@@ -1,44 +0,0 @@
-# `@crawlee/dom`
-
-Provides a framework for the parallel crawling of web pages using plain HTTP requests and [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
-
-Since `DOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.
-
-`DOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [JSDOM](https://www.npmjs.com/package/jsdom) and then invokes the user-provided [DOMCrawlerOptions.requestHandler](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestHandler) to extract page data using the `window` object.
-
-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [DOMCrawlerOptions.requestList](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestList) or [DOMCrawlerOptions.requestQueue](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestQueue) constructor options, respectively.
-
-If both [DOMCrawlerOptions.requestList](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestList) and [DOMCrawlerOptions.requestQueue](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
-
-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
-
-We can use the `preNavigationHooks` to adjust `gotOptions`:
-
-```
-preNavigationHooks: [
- (crawlingContext, gotOptions) => {
- // ...
- },
-]
-```
-
-By default, `DOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [DOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [DOMCrawlerOptions.requestHandler](https://crawlee.dev/api/dom-crawler/interface/DOMCrawlerOptions#requestHandler).
-
-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `DOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `DOMCrawler` constructor.
-
-## Example usage
-
-```javascript
-const crawler = new DOMCrawler({
- async requestHandler({ request, window }) {
- await Dataset.pushData({
- url: request.url,
- title: window.document.title,
- });
- },
-});
-
-await crawler.run([
- 'http://crawlee.dev',
-]);
-```
diff --git a/packages/dom-crawler/src/index.ts b/packages/dom-crawler/src/index.ts
deleted file mode 100644
index 8c89e6e07..000000000
--- a/packages/dom-crawler/src/index.ts
+++ /dev/null
@@ -1,2 +0,0 @@
-export * from '@crawlee/http';
-export * from './internals/dom-crawler';
diff --git a/packages/dom-crawler/.npmignore b/packages/jsdom-crawler/.npmignore
similarity index 100%
rename from packages/dom-crawler/.npmignore
rename to packages/jsdom-crawler/.npmignore
diff --git a/packages/jsdom-crawler/README.md b/packages/jsdom-crawler/README.md
new file mode 100644
index 000000000..3decfc6bf
--- /dev/null
+++ b/packages/jsdom-crawler/README.md
@@ -0,0 +1,44 @@
+# `@crawlee/jsdom`
+
+Provides a framework for the parallel crawling of web pages using plain HTTP requests and [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+Since `JSDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.
+
+`JSDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [JSDOM](https://www.npmjs.com/package/jsdom) and then invokes the user-provided [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.
+
+The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [JSDOMCrawlerOptions.requestList](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) or [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) constructor options, respectively.
+
+If both [JSDOMCrawlerOptions.requestList](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) and [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+We can use the `preNavigationHooks` to adjust `gotOptions`:
+
+```
+preNavigationHooks: [
+ (crawlingContext, gotOptions) => {
+ // ...
+ },
+]
+```
+
+By default, `JSDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [JSDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler).
+
+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `JSDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `JSDOMCrawler` constructor.
+
+## Example usage
+
+```javascript
+const crawler = new JSDOMCrawler({
+ async requestHandler({ request, window }) {
+ await Dataset.pushData({
+ url: request.url,
+ title: window.document.title,
+ });
+ },
+});
+
+await crawler.run([
+ 'http://crawlee.dev',
+]);
+```
diff --git a/packages/dom-crawler/package.json b/packages/jsdom-crawler/package.json
similarity index 98%
rename from packages/dom-crawler/package.json
rename to packages/jsdom-crawler/package.json
index 8243750a7..469008ef6 100644
--- a/packages/dom-crawler/package.json
+++ b/packages/jsdom-crawler/package.json
@@ -1,5 +1,5 @@
{
- "name": "@crawlee/dom",
+ "name": "@crawlee/jsdom",
"version": "3.0.3",
"description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
diff --git a/packages/jsdom-crawler/src/index.ts b/packages/jsdom-crawler/src/index.ts
new file mode 100644
index 000000000..2a7454461
--- /dev/null
+++ b/packages/jsdom-crawler/src/index.ts
@@ -0,0 +1,2 @@
+export * from '@crawlee/http';
+export * from './internals/jsdom-crawler';
diff --git a/packages/dom-crawler/src/internals/dom-crawler.ts b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
similarity index 74%
rename from packages/dom-crawler/src/internals/dom-crawler.ts
rename to packages/jsdom-crawler/src/internals/jsdom-crawler.ts
index 94c89e8a3..3b7866d47 100644
--- a/packages/dom-crawler/src/internals/dom-crawler.ts
+++ b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
@@ -14,57 +14,57 @@ import type { DOMWindow } from 'jsdom';
import { JSDOM } from 'jsdom';
import type { IncomingMessage } from 'http';
-export type DOMErrorHandler<
+export type JSDOMErrorHandler<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- > = ErrorHandler<DOMCrawlingContext<UserData, JSONData>>;
+ > = ErrorHandler<JSDOMCrawlingContext<UserData, JSONData>>;
-export interface DOMCrawlerOptions<
+export interface JSDOMCrawlerOptions<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- > extends HttpCrawlerOptions<DOMCrawlingContext<UserData, JSONData>> {}
+ > extends HttpCrawlerOptions<JSDOMCrawlingContext<UserData, JSONData>> {}
-export type DOMHook<
+export type JSDOMHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- > = InternalHttpHook<DOMCrawlingContext<UserData, JSONData>>;
+ > = InternalHttpHook<JSDOMCrawlingContext<UserData, JSONData>>;
-export interface DOMCrawlingContext<
+export interface JSDOMCrawlingContext<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- > extends InternalHttpCrawlingContext<UserData, JSONData, DOMCrawler> {
+ > extends InternalHttpCrawlingContext<UserData, JSONData, JSDOMCrawler> {
window: DOMWindow;
- enqueueLinks: (options?: DOMCrawlerEnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ enqueueLinks: (options?: JSDOMCrawlerEnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
}
-export type DOMRequestHandler<
+export type JSDOMRequestHandler<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
- > = RequestHandler<DOMCrawlingContext<UserData, JSONData>>;
-export interface DOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestQueue'> {}
+ > = RequestHandler<JSDOMCrawlingContext<UserData, JSONData>>;
+export interface JSDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestQueue'> {}
/**
* Provides a framework for the parallel crawling of web pages using plain HTTP requests and
- * [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation.
+ * [jsdom](https://www.npmjs.com/package/jsdom) JSDOM implementation.
* The URLs to crawl are fed either from a static list of URLs
* or from a dynamic queue of URLs enabling recursive crawling of websites.
*
- * Since `DOMCrawler` uses raw HTTP requests to download web pages,
+ * Since `JSDOMCrawler` uses raw HTTP requests to download web pages,
* it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
* to display the content, you might need to use {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler} instead,
* because it loads the pages using full-featured headless Chrome browser.
*
- * `DOMCrawler` downloads each URL using a plain HTTP request,
+ * `JSDOMCrawler` downloads each URL using a plain HTTP request,
* parses the HTML content using [JSDOM](https://www.npmjs.com/package/jsdom)
- * and then invokes the user-provided {@apilink DOMCrawlerOptions.requestHandler} to extract page data
+ * and then invokes the user-provided {@apilink JSDOMCrawlerOptions.requestHandler} to extract page data
* using the `window` object.
*
* The source URLs are represented using {@apilink Request} objects that are fed from
- * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink DOMCrawlerOptions.requestList}
- * or {@apilink DOMCrawlerOptions.requestQueue} constructor options, respectively.
+ * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink JSDOMCrawlerOptions.requestList}
+ * or {@apilink JSDOMCrawlerOptions.requestQueue} constructor options, respectively.
*
- * If both {@apilink DOMCrawlerOptions.requestList} and {@apilink DOMCrawlerOptions.requestQueue} are used,
+ * If both {@apilink JSDOMCrawlerOptions.requestList} and {@apilink JSDOMCrawlerOptions.requestQueue} are used,
* the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
* to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
@@ -80,12 +80,12 @@ export interface DOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions,
* ]
* ```
*
- * By default, `DOMCrawler` only processes web pages with the `text/html`
+ * By default, `JSDOMCrawler` only processes web pages with the `text/html`
* and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),
* and skips pages with other content types. If you want the crawler to process other content types,
- * use the {@apilink DOMCrawlerOptions.additionalMimeTypes} constructor option.
+ * use the {@apilink JSDOMCrawlerOptions.additionalMimeTypes} constructor option.
* Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.
- * For more details, see {@apilink DOMCrawlerOptions.requestHandler}.
+ * For more details, see {@apilink JSDOMCrawlerOptions.requestHandler}.
*
* New requests are only dispatched when there is enough free CPU and memory available,
* using the functionality provided by the {@apilink AutoscaledPool} class.
@@ -96,7 +96,7 @@ export interface DOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions,
* **Example usage:**
*
* ```javascript
- * const crawler = new DOMCrawler({
+ * const crawler = new JSDOMCrawler({
* async requestHandler({ request, window }) {
* await Dataset.pushData({
* url: request.url,
@@ -111,8 +111,8 @@ export interface DOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions,
* ```
* @category Crawlers
*/
-export class DOMCrawler extends HttpCrawler<DOMCrawlingContext> {
- protected override async _parseHTML(response: IncomingMessage, isXml: boolean, crawlingContext: DOMCrawlingContext) {
+export class JSDOMCrawler extends HttpCrawler<JSDOMCrawlingContext> {
+ protected override async _parseHTML(response: IncomingMessage, isXml: boolean, crawlingContext: JSDOMCrawlingContext) {
const body = await concatStreamToBuffer(response);
const { window } = new JSDOM(body, {
@@ -125,7 +125,7 @@ export class DOMCrawler extends HttpCrawler<DOMCrawlingContext> {
get body() {
return window.document.documentElement.outerHTML;
},
- enqueueLinks: async (enqueueOptions?: DOMCrawlerEnqueueLinksOptions) => {
+ enqueueLinks: async (enqueueOptions?: JSDOMCrawlerEnqueueLinksOptions) => {
return domCrawlerEnqueueLinks({
options: enqueueOptions,
window,
@@ -139,7 +139,7 @@ export class DOMCrawler extends HttpCrawler<DOMCrawlingContext> {
}
interface EnqueueLinksInternalOptions {
- options?: DOMCrawlerEnqueueLinksOptions;
+ options?: JSDOMCrawlerEnqueueLinksOptions;
window: DOMWindow | null;
requestQueue: RequestQueue;
originalRequestUrl: string;
@@ -149,7 +149,7 @@ interface EnqueueLinksInternalOptions {
/** @internal */
export async function domCrawlerEnqueueLinks({ options, window, requestQueue, originalRequestUrl, finalRequestUrl }: EnqueueLinksInternalOptions) {
if (!window) {
- throw new Error('Cannot enqueue links because the DOM is not available.');
+ throw new Error('Cannot enqueue links because the JSDOM is not available.');
}
const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
@@ -193,15 +193,15 @@ function extractUrlsFromWindow(window: DOMWindow, selector: string, baseUrl: str
/**
* Creates new {@apilink Router} instance that works based on request labels.
- * This instance can then serve as a `requestHandler` of your {@apilink DOMCrawler}.
- * Defaults to the {@apilink DOMCrawlingContext}.
+ * This instance can then serve as a `requestHandler` of your {@apilink JSDOMCrawler}.
+ * Defaults to the {@apilink JSDOMCrawlingContext}.
*
- * > Serves as a shortcut for using `Router.create<DOMCrawlingContext>()`.
+ * > Serves as a shortcut for using `Router.create<JSDOMCrawlingContext>()`.
*
* ```ts
- * import { DOMCrawler, createDOMRouter } from 'crawlee';
+ * import { JSDOMCrawler, createJSDOMRouter } from 'crawlee';
*
- * const router = createDOMRouter();
+ * const router = createJSDOMRouter();
* router.addHandler('label-a', async (ctx) => {
* ctx.log.info('...');
* });
@@ -209,12 +209,12 @@ function extractUrlsFromWindow(window: DOMWindow, selector: string, baseUrl: str
* ctx.log.info('...');
* });
*
- * const crawler = new DOMCrawler({
+ * const crawler = new JSDOMCrawler({
* requestHandler: router,
* });
* await crawler.run();
* ```
*/
-export function createDOMRouter<Context extends DOMCrawlingContext = DOMCrawlingContext>() {
+export function createJSDOMRouter<Context extends JSDOMCrawlingContext = JSDOMCrawlingContext>() {
return Router.create<Context>();
}
diff --git a/packages/dom-crawler/tsconfig.build.json b/packages/jsdom-crawler/tsconfig.build.json
similarity index 100%
rename from packages/dom-crawler/tsconfig.build.json
rename to packages/jsdom-crawler/tsconfig.build.json
diff --git a/packages/dom-crawler/tsconfig.json b/packages/jsdom-crawler/tsconfig.json
similarity index 100%
rename from packages/dom-crawler/tsconfig.json
rename to packages/jsdom-crawler/tsconfig.json
diff --git a/test/core/crawlers/dom_crawler.test.ts b/test/core/crawlers/dom_crawler.test.ts
index d083b353a..abcd7c9f4 100644
--- a/test/core/crawlers/dom_crawler.test.ts
+++ b/test/core/crawlers/dom_crawler.test.ts
@@ -1,4 +1,4 @@
-import { DOMCrawler } from '@crawlee/dom';
+import { JSDOMCrawler } from '@crawlee/jsdom';
import type { AddressInfo } from 'node:net';
import http from 'node:http';
import { MemoryStorageEmulator } from 'test/shared/MemoryStorageEmulator';
@@ -45,7 +45,7 @@ afterAll(async () => {
test('works', async () => {
const results: string[] = [];
- const crawler = new DOMCrawler({
+ const crawler = new JSDOMCrawler({
maxRequestRetries: 0,
requestHandler: ({ window }) => {
results.push(window.document.title, window.document.querySelector('p').textContent);
diff --git a/test/e2e/dom-default-ts/actor/.gitignore b/test/e2e/jsdom-default-ts/actor/.gitignore
similarity index 100%
rename from test/e2e/dom-default-ts/actor/.gitignore
rename to test/e2e/jsdom-default-ts/actor/.gitignore
diff --git a/test/e2e/dom-default-ts/actor/Dockerfile b/test/e2e/jsdom-default-ts/actor/Dockerfile
similarity index 100%
rename from test/e2e/dom-default-ts/actor/Dockerfile
rename to test/e2e/jsdom-default-ts/actor/Dockerfile
diff --git a/test/e2e/dom-default-ts/actor/apify.json b/test/e2e/jsdom-default-ts/actor/apify.json
similarity index 63%
rename from test/e2e/dom-default-ts/actor/apify.json
rename to test/e2e/jsdom-default-ts/actor/apify.json
index f9b4f3874..dfa4a1d45 100644
--- a/test/e2e/dom-default-ts/actor/apify.json
+++ b/test/e2e/jsdom-default-ts/actor/apify.json
@@ -1,5 +1,5 @@
{
- "name": "test-dom-default-ts",
+ "name": "test-jsdom-default-ts",
"version": "0.0",
"buildTag": "latest",
"env": null
diff --git a/test/e2e/dom-default-ts/actor/main.ts b/test/e2e/jsdom-default-ts/actor/main.ts
similarity index 88%
rename from test/e2e/dom-default-ts/actor/main.ts
rename to test/e2e/jsdom-default-ts/actor/main.ts
index 2d36c2c47..fc7821750 100644
--- a/test/e2e/dom-default-ts/actor/main.ts
+++ b/test/e2e/jsdom-default-ts/actor/main.ts
@@ -1,5 +1,5 @@
import { Actor } from 'apify';
-import { DOMCrawler, Dataset } from '@crawlee/dom';
+import { JSDOMCrawler, Dataset } from '@crawlee/jsdom';
import { ApifyStorageLocal } from '@apify/storage-local';
if (process.env.STORAGE_IMPLEMENTATION === 'LOCAL') {
@@ -8,7 +8,7 @@ if (process.env.STORAGE_IMPLEMENTATION === 'LOCAL') {
await Actor.init();
}
-const crawler = new DOMCrawler();
+const crawler = new JSDOMCrawler();
crawler.router.addDefaultHandler(async ({ window, enqueueLinks, request, log }) => {
const { url } = request;
diff --git a/test/e2e/dom-default-ts/actor/package.json b/test/e2e/jsdom-default-ts/actor/package.json
similarity index 84%
rename from test/e2e/dom-default-ts/actor/package.json
rename to test/e2e/jsdom-default-ts/actor/package.json
index 404012934..0070966f3 100644
--- a/test/e2e/dom-default-ts/actor/package.json
+++ b/test/e2e/jsdom-default-ts/actor/package.json
@@ -1,14 +1,14 @@
{
- "name": "test-dom-default-ts",
+ "name": "test-jsdom-default-ts",
"version": "0.0.1",
- "description": "DOM Crawler Test - TypeScript",
+ "description": "JSDOM Crawler Test - TypeScript",
"dependencies": {
"apify": "^3.0.0-beta.75",
"@apify/storage-local": "^2.1.0",
"@crawlee/basic": "file:./packages/basic-crawler",
"@crawlee/browser-pool": "file:./packages/browser-pool",
"@crawlee/http": "file:./packages/http-crawler",
- "@crawlee/dom": "file:./packages/dom-crawler",
+ "@crawlee/jsdom": "file:./packages/jsdom-crawler",
"@crawlee/core": "file:./packages/core",
"@crawlee/memory-storage": "file:./packages/memory-storage",
"@crawlee/types": "file:./packages/types",
diff --git a/test/e2e/dom-default-ts/actor/tsconfig.json b/test/e2e/jsdom-default-ts/actor/tsconfig.json
similarity index 100%
rename from test/e2e/dom-default-ts/actor/tsconfig.json
rename to test/e2e/jsdom-default-ts/actor/tsconfig.json
diff --git a/test/e2e/dom-default-ts/test.mjs b/test/e2e/jsdom-default-ts/test.mjs
similarity index 100%
rename from test/e2e/dom-default-ts/test.mjs
rename to test/e2e/jsdom-default-ts/test.mjs
diff --git a/tsconfig.json b/tsconfig.json
index aadee6a9f..931ddc4f8 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -7,7 +7,7 @@
"@crawlee/basic": ["packages/basic-crawler/src"],
"@crawlee/browser": ["packages/browser-crawler/src"],
"@crawlee/http": ["packages/http-crawler/src"],
- "@crawlee/dom": ["packages/dom-crawler/src"],
+ "@crawlee/jsdom": ["packages/jsdom-crawler/src"],
"@crawlee/cheerio": ["packages/cheerio-crawler/src"],
"@crawlee/playwright": ["packages/playwright-crawler/src"],
"@crawlee/puppeteer": ["packages/puppeteer-crawler/src"],
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index dd4a4a379..3d3bf43ef 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -14,7 +14,7 @@ const packages = [
'cheerio-crawler',
'puppeteer-crawler',
'playwright-crawler',
- 'dom-crawler',
+ 'jsdom-crawler',
'memory-storage',
'utils',
'types',
@@ -22,7 +22,7 @@ const packages = [
const packagesOrder = [
'@crawlee/core',
'@crawlee/cheerio',
- '@crawlee/dom',
+ '@crawlee/jsdom',
'@crawlee/playwright',
'@crawlee/puppeteer',
'@crawlee/basic',
diff --git a/website/sidebars.js b/website/sidebars.js
index 5c6d229d2..2ba457076 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -39,7 +39,7 @@ module.exports = {
'guides/session-management',
'guides/scaling-crawlers',
'guides/avoid-blocking',
- 'guides/dom-crawler-guide',
+ 'guides/jsdom-crawler-guide',
'guides/got-scraping',
'guides/typescript-project',
'guides/docker-images',