Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: add LinkeDOMCrawler (#1907)
Pavel 'Strajk' Dolecek1c69560fe7ef
diff --git a/docs/examples/tsconfig.json b/docs/examples/tsconfig.jsonindex e1b8e5b1a..62b847082 100644--- a/docs/examples/tsconfig.json+++ b/docs/examples/tsconfig.json@@ -7,6 +7,7 @@"@crawlee/basic": ["packages/basic-crawler/src"],"@crawlee/browser": ["packages/browser-crawler/src"],"@crawlee/http": ["packages/http-crawler/src"],+ "@crawlee/linkedom": ["packages/linkedom-crawler/src"],"@crawlee/jsdom": ["packages/jsdom-crawler/src"],"@crawlee/cheerio": ["packages/cheerio-crawler/src"],"@crawlee/playwright": ["packages/playwright-crawler/src"],diff --git a/jest.config.js b/jest.config.jsindex f763b8745..e47047973 100644--- a/jest.config.js+++ b/jest.config.js@@ -17,6 +17,7 @@ module.exports = {'^@crawlee/basic$': '<rootDir>/packages/basic-crawler/src','^@crawlee/browser$': '<rootDir>/packages/browser-crawler/src','^@crawlee/http$': '<rootDir>/packages/http-crawler/src',+ '^@crawlee/linkedom$': '<rootDir>/packages/linkedom-crawler/src','^@crawlee/jsdom$': '<rootDir>/packages/jsdom-crawler/src','^@crawlee/cheerio$': '<rootDir>/packages/cheerio-crawler/src','^@crawlee/playwright$': '<rootDir>/packages/playwright-crawler/src',diff --git a/packages/core/README.md b/packages/core/README.mdindex 6e88d36f6..a1f73373f 100644--- a/packages/core/README.md+++ b/packages/core/README.md@@ -8,6 +8,7 @@ The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of sever- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/linkedom`](https://crawlee.dev/api/jsdom-crawler): exports `LinkeDOMCrawler`- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))diff --git a/packages/linkedom-crawler/.npmignore b/packages/linkedom-crawler/.npmignorenew file mode 100644index 000000000..ada367d72--- /dev/null+++ b/packages/linkedom-crawler/.npmignore@@ -0,0 +1,6 @@+node_modules+src+test+coverage+apify_storage+tsconfig.*diff --git a/packages/linkedom-crawler/CHANGELOG.md b/packages/linkedom-crawler/CHANGELOG.mdnew file mode 100644index 000000000..e69de29bbdiff --git a/packages/linkedom-crawler/README.md b/packages/linkedom-crawler/README.mdnew file mode 100644index 000000000..b3b506cd2--- /dev/null+++ b/packages/linkedom-crawler/README.md@@ -0,0 +1,46 @@+# `@crawlee/linkedom`++<!-- Note for editors: This README should be kept consistent with JSDOMCrawler one -->++Provides a framework for the parallel crawling of web pages using plain HTTP requests and [linkedom](https://www.npmjs.com/package/linkedom) DOM implementation. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.++Since `LinkeDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.++`LinkeDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [LinkeDOM](https://www.npmjs.com/package/linkedom) and then invokes the user-provided [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.++The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) or [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) constructor options, respectively.++If both [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) and [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.++The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.++We can use the `preNavigationHooks` to adjust `gotOptions`:++```+preNavigationHooks: [+ (crawlingContext, gotOptions) => {+ // ...+ },+]+```++By default, `LinkeDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [LinkeDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler).++New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `LinkeDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `LinkeDOMCrawler` constructor.++## Example usage++```javascript+const crawler = new LinkeDOMCrawler({+ async requestHandler({ request, window }) {+ await Dataset.pushData({+ url: request.url,+ title: window.document.title,+ });+ },+});++await crawler.run([+ 'http://crawlee.dev',+]);+```diff --git a/packages/linkedom-crawler/package.json b/packages/linkedom-crawler/package.jsonnew file mode 100644index 000000000..8cf52fe40--- /dev/null+++ b/packages/linkedom-crawler/package.json@@ -0,0 +1,64 @@+{+ "name": "@crawlee/linkedom",+ "version": "3.3.3",+ "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",+ "engines": {+ "node": ">=16.0.0"+ },+ "main": "./dist/index.js",+ "module": "./dist/index.mjs",+ "types": "./dist/index.d.ts",+ "exports": {+ ".": {+ "import": "./dist/index.mjs",+ "require": "./dist/index.js",+ "types": "./dist/index.d.ts"+ },+ "./package.json": "./package.json"+ },+ "keywords": [+ "apify",+ "headless",+ "chrome",+ "puppeteer",+ "crawler",+ "scraper"+ ],+ "author": {+ "name": "Apify",+ "email": "[email protected]",+ "url": "https://apify.com"+ },+ "contributors": [+ "Jan Curn <[email protected]>",+ "Marek Trunkat <[email protected]>",+ "Ondra Urban <[email protected]>"+ ],+ "license": "Apache-2.0",+ "repository": {+ "type": "git",+ "url": "git+https://github.com/apify/crawlee"+ },+ "bugs": {+ "url": "https://github.com/apify/crawlee/issues"+ },+ "homepage": "https://crawlee.dev",+ "scripts": {+ "build": "yarn clean && yarn compile && yarn copy",+ "clean": "rimraf ./dist",+ "compile": "tsc -p tsconfig.build.json && gen-esm-wrapper ./dist/index.js ./dist/index.mjs",+ "copy": "ts-node -T ../../scripts/copy.ts"+ },+ "publishConfig": {+ "access": "public"+ },+ "dependencies": {+ "@apify/timeout": "^0.3.0",+ "@apify/utilities": "^2.1.4",+ "@crawlee/http": "^3.3.1",+ "@crawlee/types": "^3.3.1",+ "linkedom": "^0.14.25",+ "ow": "^0.28.2",+ "tslib": "^2.4.0"+ }+}diff --git a/packages/linkedom-crawler/src/index.ts b/packages/linkedom-crawler/src/index.tsnew file mode 100644index 000000000..c52d14dcb--- /dev/null+++ b/packages/linkedom-crawler/src/index.ts@@ -0,0 +1,2 @@+export * from '@crawlee/http';+export * from './internals/linkedom-crawler';diff --git a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts b/packages/linkedom-crawler/src/internals/linkedom-crawler.tsnew file mode 100644index 000000000..67eea55fe--- /dev/null+++ b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts@@ -0,0 +1,233 @@+import type * as cheerio from 'cheerio';+import type {+ HttpCrawlerOptions,+ InternalHttpCrawlingContext,+ InternalHttpHook,+ ErrorHandler,+ RequestHandler,+ EnqueueLinksOptions,+ RequestQueue,+} from '@crawlee/http';+import { HttpCrawler, enqueueLinks, Router, resolveBaseUrlForEnqueueLinksFiltering, tryAbsoluteURL } from '@crawlee/http';+import type { Dictionary, GetUserDataFromRequest, RouterRoutes } from '@crawlee/core';+import { concatStreamToBuffer } from '@apify/utilities';+import { DOMParser } from 'linkedom/cached';+import type { IncomingMessage } from 'http';++export type LinkeDOMErrorHandler<+ UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ > = ErrorHandler<LinkeDOMCrawlingContext<UserData, JSONData>>;++export interface LinkeDOMCrawlerOptions<+ UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ > extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>> {}++export interface LinkeDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestQueue'> {}++export type LinkeDOMHook<+ UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ > = InternalHttpHook<LinkeDOMCrawlingContext<UserData, JSONData>>;++export interface LinkeDOMCrawlingContext<+ UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ > extends InternalHttpCrawlingContext<UserData, JSONData, LinkeDOMCrawler> {+ window: Window;++ /**+ * Returns Cheerio handle, allowing to work with the data same way as with {@apilink CheerioCrawler}.+ *+ * **Example usage:**+ * ```javascript+ * async requestHandler({ parseWithCheerio }) {+ * const $ = await parseWithCheerio();+ * const title = $('title').text();+ * });+ * ```+ */+ parseWithCheerio(): Promise<cheerio.CheerioAPI>;+}++export type LinkeDOMRequestHandler<+ UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler+ > = RequestHandler<LinkeDOMCrawlingContext<UserData, JSONData>>;++/**+ * Provides a framework for the parallel crawling of web pages using plain HTTP requests and+ * [linkedom](https://www.npmjs.com/package/linkedom) LinkeDOM implementation.+ * The URLs to crawl are fed either from a static list of URLs+ * or from a dynamic queue of URLs enabling recursive crawling of websites.+ *+ * Since `LinkeDOMCrawler` uses raw HTTP requests to download web pages,+ * it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript+ * to display the content, you might need to use {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler} instead,+ * because it loads the pages using full-featured headless Chrome browser.+ *+ * **Limitation**:+ * This crawler does not support proxies and cookies yet (each open starts with empty cookie store), and the user agent is always set to `Chrome`.+ *+ * `LinkeDOMCrawler` downloads each URL using a plain HTTP request,+ * parses the HTML content using [LinkeDOM](https://www.npmjs.com/package/linkedom)+ * and then invokes the user-provided {@apilink LinkeDOMCrawlerOptions.requestHandler} to extract page data+ * using the `window` object.+ *+ * The source URLs are represented using {@apilink Request} objects that are fed from+ * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink LinkeDOMCrawlerOptions.requestList}+ * or {@apilink LinkeDOMCrawlerOptions.requestQueue} constructor options, respectively.+ *+ * If both {@apilink LinkeDOMCrawlerOptions.requestList} and {@apilink LinkeDOMCrawlerOptions.requestQueue} are used,+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them+ * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.+ *+ * The crawler finishes when there are no more {@apilink Request} objects to crawl.+ *+ * We can use the `preNavigationHooks` to adjust `gotOptions`:+ *+ * ```+ * preNavigationHooks: [+ * (crawlingContext, gotOptions) => {+ * // ...+ * },+ * ]+ * ```+ *+ * By default, `LinkeDOMCrawler` only processes web pages with the `text/html`+ * and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),+ * and skips pages with other content types. If you want the crawler to process other content types,+ * use the {@apilink LinkeDOMCrawlerOptions.additionalMimeTypes} constructor option.+ * Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.+ * For more details, see {@apilink LinkeDOMCrawlerOptions.requestHandler}.+ *+ * New requests are only dispatched when there is enough free CPU and memory available,+ * using the functionality provided by the {@apilink AutoscaledPool} class.+ * All {@apilink AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`+ * parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`+ * {@apilink AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.+ *+ * **Example usage:**+ *+ * ```javascript+ * const crawler = new LinkeDOMCrawler({+ * async requestHandler({ request, window }) {+ * await Dataset.pushData({+ * url: request.url,+ * title: window.document.title,+ * });+ * },+ * });+ *+ * await crawler.run([+ * 'http://crawlee.dev',+ * ]);+ * ```+ * @category Crawlers+ */++export class LinkeDOMCrawler extends HttpCrawler<LinkeDOMCrawlingContext> {+ private static parser = new DOMParser();++ protected override async _parseHTML(response: IncomingMessage, isXml: boolean, crawlingContext: LinkeDOMCrawlingContext) {+ const body = await concatStreamToBuffer(response);++ const document = LinkeDOMCrawler.parser.parseFromString(body.toString(), isXml ? 'text/html' : 'text/html');++ return {+ window: document.defaultView,+ get body() {+ return document.documentElement.outerHTML;+ },+ enqueueLinks: async (enqueueOptions?: LinkeDOMCrawlerEnqueueLinksOptions) => {+ return linkedomCrawlerEnqueueLinks({+ options: enqueueOptions,+ window: document.defaultView,+ requestQueue: await this.getRequestQueue(),+ originalRequestUrl: crawlingContext.request.url,+ finalRequestUrl: crawlingContext.request.loadedUrl,+ });+ },+ };+ }+}++interface EnqueueLinksInternalOptions {+ options?: LinkeDOMCrawlerEnqueueLinksOptions;+ window: Window | null;+ requestQueue: RequestQueue;+ originalRequestUrl: string;+ finalRequestUrl?: string;+}++/** @internal */+export async function linkedomCrawlerEnqueueLinks({ options, window, requestQueue, originalRequestUrl, finalRequestUrl }: EnqueueLinksInternalOptions) {+ if (!window) {+ throw new Error('Cannot enqueue links because the DOM is not available.');+ }++ const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({+ enqueueStrategy: options?.strategy,+ finalRequestUrl,+ originalRequestUrl,+ userProvidedBaseUrl: options?.baseUrl,+ });++ const urls = extractUrlsFromWindow(window, options?.selector ?? 'a', options?.baseUrl ?? finalRequestUrl ?? originalRequestUrl);++ return enqueueLinks({+ requestQueue,+ urls,+ baseUrl,+ ...options,+ });+}++/**+ * Extracts URLs from a given Window object.+ * @ignore+ */+function extractUrlsFromWindow(window: Window, selector: string, baseUrl: string): string[] {+ return Array.from(window.document.querySelectorAll(selector))+ .map((e: any) => e.href)+ .filter((href) => href !== undefined && href !== '')+ .map((href: string | undefined) => {+ if (href === undefined) {+ return undefined;+ }+ return tryAbsoluteURL(href, baseUrl);+ })+ .filter((href) => href !== undefined && href !== '') as string[];+}++/**+ * Creates new {@apilink Router} instance that works based on request labels.+ * This instance can then serve as a `requestHandler` of your {@apilink LinkeDOMCrawler}.+ * Defaults to the {@apilink LinkeDOMCrawlingContext}.+ *+ * > Serves as a shortcut for using `Router.create<LinkeDOMCrawlingContext>()`.+ *+ * ```ts+ * import { LinkeDOMCrawler, createLinkeDOMRouter } from 'crawlee';+ *+ * const router = createLinkeDOMRouter();+ * router.addHandler('label-a', async (ctx) => {+ * ctx.log.info('...');+ * });+ * router.addDefaultHandler(async (ctx) => {+ * ctx.log.info('...');+ * });+ *+ * const crawler = new LinkeDOMCrawler({+ * requestHandler: router,+ * });+ * await crawler.run();+ * ```+ */+export function createLinkeDOMRouter<+ Context extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext,+ UserData extends Dictionary = GetUserDataFromRequest<Context['request']>,+>(routes?: RouterRoutes<Context, UserData>) {+ return Router.create<Context>(routes);+}diff --git a/packages/linkedom-crawler/tsconfig.build.json b/packages/linkedom-crawler/tsconfig.build.jsonnew file mode 100644index 000000000..856db0f21--- /dev/null+++ b/packages/linkedom-crawler/tsconfig.build.json@@ -0,0 +1,7 @@+{+ "extends": "../../tsconfig.build.json",+ "compilerOptions": {+ "outDir": "./dist"+ },+ "include": ["src/**/*"]+}diff --git a/packages/linkedom-crawler/tsconfig.json b/packages/linkedom-crawler/tsconfig.jsonnew file mode 100644index 000000000..52d43eaaa--- /dev/null+++ b/packages/linkedom-crawler/tsconfig.json@@ -0,0 +1,4 @@+{+ "extends": "../../tsconfig.json",+ "include": ["src/**/*"]+}diff --git a/test/e2e/linkedom-default-ts/actor/.actor/actor.json b/test/e2e/linkedom-default-ts/actor/.actor/actor.jsonnew file mode 100644index 000000000..f83df08dc--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/.actor/actor.json@@ -0,0 +1,7 @@+{+ "actorSpecification": 1,+ "name": "test-linkedom-default-ts",+ "version": "0.0",+ "buildTag": "latest",+ "env": null+}diff --git a/test/e2e/linkedom-default-ts/actor/.gitignore b/test/e2e/linkedom-default-ts/actor/.gitignorenew file mode 100644index 000000000..f2fc11c72--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/.gitignore@@ -0,0 +1,11 @@+.idea+.DS_Store+node_modules+package-lock.json+apify_storage+crawlee_storage+storage+main.d.ts+main.d.ts.map+main.js+main.js.mapdiff --git a/test/e2e/linkedom-default-ts/actor/Dockerfile b/test/e2e/linkedom-default-ts/actor/Dockerfilenew file mode 100644index 000000000..4bde52b93--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/Dockerfile@@ -0,0 +1,28 @@+# using multistage build, as we need dev deps to build the TS source code+FROM apify/actor-node:18-beta AS builder++# copy all files, install all dependencies (including dev deps) and build the project+COPY . ./+RUN npm install --include=dev \+ && npm run build++# create final image+FROM apify/actor-node:18-beta+# copy only necessary files+COPY --from=builder /usr/src/app/packages ./packages+COPY --from=builder /usr/src/app/package.json ./+COPY --from=builder /usr/src/app/main.js ./++# install only prod deps+RUN npm --quiet set progress=false \+ && npm install --only=prod --no-optional \+ && npm update \+ && echo "Installed NPM packages:" \+ && (npm list --only=prod --no-optional --all || true) \+ && echo "Node.js version:" \+ && node --version \+ && echo "NPM version:" \+ && npm --version++# run compiled code+CMD npm run start:proddiff --git a/test/e2e/linkedom-default-ts/actor/main.ts b/test/e2e/linkedom-default-ts/actor/main.tsnew file mode 100644index 000000000..4a54374a1--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/main.ts@@ -0,0 +1,27 @@+import { Actor } from 'apify';+import { LinkeDOMCrawler, Dataset } from '@crawlee/linkedom';++if (process.env.STORAGE_IMPLEMENTATION === 'LOCAL') {+ const { ApifyStorageLocal } = await import('@apify/storage-local');+ await Actor.init({ storage: new ApifyStorageLocal() });+} else {+ await Actor.init();+}++const crawler = new LinkeDOMCrawler();++crawler.router.addDefaultHandler(async ({ document, enqueueLinks, request, log }) => {+ const { url } = request;+ await enqueueLinks({+ globs: ['https://crawlee.dev/docs/**'],+ });++ const pageTitle = document.title;+ log.info(`URL: ${url} TITLE: ${pageTitle}`);++ await Dataset.pushData({ url, pageTitle });+});++await crawler.run(['https://crawlee.dev/docs/quick-start']);++await Actor.exit({ exit: Actor.isAtHome() });diff --git a/test/e2e/linkedom-default-ts/actor/package.json b/test/e2e/linkedom-default-ts/actor/package.jsonnew file mode 100644index 000000000..3381e4d37--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/package.json@@ -0,0 +1,35 @@+{+ "name": "test-linkedom-default-ts",+ "version": "0.0.1",+ "description": "LinkeDOM Crawler Test - TypeScript",+ "dependencies": {+ "apify": "next",+ "@apify/storage-local": "^2.1.0",+ "@crawlee/basic": "file:./packages/basic-crawler",+ "@crawlee/browser-pool": "file:./packages/browser-pool",+ "@crawlee/http": "file:./packages/http-crawler",+ "@crawlee/linkedom": "file:./packages/linkedom-crawler",+ "@crawlee/core": "file:./packages/core",+ "@crawlee/memory-storage": "file:./packages/memory-storage",+ "@crawlee/types": "file:./packages/types",+ "@crawlee/utils": "file:./packages/utils"+ },+ "overrides": {+ "apify": {+ "@crawlee/core": "file:./packages/core",+ "@crawlee/types": "file:./packages/types",+ "@crawlee/utils": "file:./packages/utils"+ }+ },+ "devDependencies": {+ "@apify/tsconfig": "^0.1.0",+ "typescript": "4.7.4"+ },+ "scripts": {+ "start": "tsc && node main.js",+ "start:prod": "node main.js",+ "build": "tsc"+ },+ "type": "module",+ "license": "ISC"+}diff --git a/test/e2e/linkedom-default-ts/actor/tsconfig.json b/test/e2e/linkedom-default-ts/actor/tsconfig.jsonnew file mode 100644index 000000000..741845092--- /dev/null+++ b/test/e2e/linkedom-default-ts/actor/tsconfig.json@@ -0,0 +1,11 @@+{+ "extends": "@apify/tsconfig",+ "compilerOptions": {+ "module": "ES2022",+ "target": "ES2022",+ "lib": ["DOM"]+ },+ "include": [+ "./**/*.ts"+ ]+}diff --git a/test/e2e/linkedom-default-ts/test.mjs b/test/e2e/linkedom-default-ts/test.mjsnew file mode 100644index 000000000..bf2015b4e--- /dev/null+++ b/test/e2e/linkedom-default-ts/test.mjs@@ -0,0 +1,10 @@+import { initialize, getActorTestDir, runActor, expect, validateDataset } from '../tools.mjs';++const testActorDirname = getActorTestDir(import.meta.url);+await initialize(testActorDirname);++const { stats, datasetItems } = await runActor(testActorDirname);++await expect(stats.requestsFinished > 40, 'All requests finished');+await expect(datasetItems.length > 40, 'Number of dataset items');+await expect(validateDataset(datasetItems, ['url', 'pageTitle']), 'Dataset items validation');diff --git a/tsconfig.json b/tsconfig.jsonindex 931ddc4f8..975f7708a 100644--- a/tsconfig.json+++ b/tsconfig.json@@ -7,6 +7,7 @@"@crawlee/basic": ["packages/basic-crawler/src"],"@crawlee/browser": ["packages/browser-crawler/src"],"@crawlee/http": ["packages/http-crawler/src"],+ "@crawlee/linkedom": ["packages/linkedom-crawler/src"],"@crawlee/jsdom": ["packages/jsdom-crawler/src"],"@crawlee/cheerio": ["packages/cheerio-crawler/src"],"@crawlee/playwright": ["packages/playwright-crawler/src"],diff --git a/yarn.lock b/yarn.lockindex 1fc87222c..9e1d2e6aa 100644--- a/yarn.lock+++ b/yarn.lock@@ -849,7 +849,7 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/http@^3.3.3, @crawlee/http@workspace:packages/http-crawler":+"@crawlee/http@^3.3.1, @crawlee/http@^3.3.3, @crawlee/http@workspace:packages/http-crawler":version: 0.0.0-use.localresolution: "@crawlee/http@workspace:packages/http-crawler"dependencies:@@ -885,6 +885,20 @@ __metadata:languageName: unknownlinkType: soft+"@crawlee/linkedom@workspace:packages/linkedom-crawler":+ version: 0.0.0-use.local+ resolution: "@crawlee/linkedom@workspace:packages/linkedom-crawler"+ dependencies:+ "@apify/timeout": ^0.3.0+ "@apify/utilities": ^2.1.4+ "@crawlee/http": ^3.3.1+ "@crawlee/types": ^3.3.1+ linkedom: ^0.14.25+ ow: ^0.28.2+ tslib: ^2.4.0+ languageName: unknown+ linkType: soft+"@crawlee/memory-storage@^3.3.3, @crawlee/memory-storage@workspace:packages/memory-storage":version: 0.0.0-use.localresolution: "@crawlee/memory-storage@workspace:packages/memory-storage"@@ -1015,7 +1029,7 @@ __metadata:languageName: unknownlinkType: soft-"@crawlee/types@^3.3.0, @crawlee/types@^3.3.3, @crawlee/types@workspace:packages/types":+"@crawlee/types@^3.3.0, @crawlee/types@^3.3.1, @crawlee/types@^3.3.3, @crawlee/types@workspace:packages/types":version: 0.0.0-use.localresolution: "@crawlee/types@workspace:packages/types"dependencies:@@ -4673,6 +4687,13 @@ __metadata:languageName: nodelinkType: hard+"cssom@npm:^0.5.0":+ version: 0.5.0+ resolution: "cssom@npm:0.5.0"+ checksum: 823471aa30091c59e0a305927c30e7768939b6af70405808f8d2ce1ca778cddcb24722717392438329d1691f9a87cb0183b64b8d779b56a961546d54854fde01+ languageName: node+ linkType: hard+"cssstyle@npm:^3.0.0":version: 3.0.0resolution: "cssstyle@npm:3.0.0"@@ -6898,6 +6919,13 @@ __metadata:languageName: nodelinkType: hard+"html-escaper@npm:^3.0.3":+ version: 3.0.3+ resolution: "html-escaper@npm:3.0.3"+ checksum: a2678be42c15d2ef6e629775dac0925a729f4615c6593db8358b9262c7565c4627134987c00f548eb4eb76cbc3b3392f78475cd02b022f8ae7aeb9a88280831b+ languageName: node+ linkType: hard+"htmlparser2@npm:^8.0.1":version: 8.0.2resolution: "htmlparser2@npm:8.0.2"@@ -8628,6 +8656,19 @@ __metadata:languageName: nodelinkType: hard+"linkedom@npm:^0.14.25":+ version: 0.14.26+ resolution: "linkedom@npm:0.14.26"+ dependencies:+ css-select: ^5.1.0+ cssom: ^0.5.0+ html-escaper: ^3.0.3+ htmlparser2: ^8.0.1+ uhyphen: ^0.2.0+ checksum: 828415ac9f848efb307f444c47b3966383f782bd6cc8e52cf839f6a901d3ef2dc56da11aee3454794b345f08b04960152a3a339753295cd7b72084608f7314a0+ languageName: node+ linkType: hard+"lint-staged@npm:^13.0.3":version: 13.2.2resolution: "lint-staged@npm:13.2.2"@@ -12768,6 +12809,13 @@ __metadata:languageName: nodelinkType: hard+"uhyphen@npm:^0.2.0":+ version: 0.2.0+ resolution: "uhyphen@npm:0.2.0"+ checksum: b946e2c17989f50412d7cc07521a199a1edc52e0320813c06ef5eee21570cd46bbfeb98e82288f21150144a3ef9d4e99395533ab1c918030e5224f3e247cbe57+ languageName: node+ linkType: hard+"unbox-primitive@npm:^1.0.2":version: 1.0.2resolution: "unbox-primitive@npm:1.0.2"