Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

feat: split `enqueueLinks` into `extractLinks` + `addRequests` (#4010)

Jindřich Bär6bcd06e66845
diff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl_all_links.mdx
index 371b834ca..efd2786af 100644
--- a/docs/examples/crawl_all_links.mdx
+++ b/docs/examples/crawl_all_links.mdx
@@ -17,7 +17,7 @@ as the crawler navigates from page to page. This example can also be used to fin
:::tip
-If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the <ApiLink to="core/interface/EnqueueLinksOptions#strategy">`strategy`</ApiLink>
+If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the <ApiLink to="core/interface/EnqueueUrlsOptions#strategy">`strategy`</ApiLink>
option. You can find more info about this option in the [`Crawl relative links`](./crawl-relative-links) examples.
:::
diff --git a/docs/examples/crawl_some_links.mdx b/docs/examples/crawl_some_links.mdx
index b9b7cf85f..11c656679 100644
--- a/docs/examples/crawl_some_links.mdx
+++ b/docs/examples/crawl_some_links.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';
-This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="core/interface/EnqueueLinksOptions#include">`include`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.
+This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="core/interface/EnqueueUrlsOptions#include">`include`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.
<RunnableCodeBlock className="language-js" type="cheerio">
{CrawlSource}
diff --git a/docs/examples/crawler-plugins/playwright-extra.ts b/docs/examples/crawler-plugins/playwright-extra.ts
index 8cb6f7538..2121fc3df 100644
--- a/docs/examples/crawler-plugins/playwright-extra.ts
+++ b/docs/examples/crawler-plugins/playwright-extra.ts
@@ -57,7 +57,7 @@ const crawler = new PlaywrightCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/crawler-plugins/puppeteer-extra.ts b/docs/examples/crawler-plugins/puppeteer-extra.ts
index a0a197f38..058119719 100644
--- a/docs/examples/crawler-plugins/puppeteer-extra.ts
+++ b/docs/examples/crawler-plugins/puppeteer-extra.ts
@@ -55,7 +55,7 @@ const crawler = new PuppeteerCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/playwright_crawler.ts b/docs/examples/playwright_crawler.ts
index 9595a021b..e8861322c 100644
--- a/docs/examples/playwright_crawler.ts
+++ b/docs/examples/playwright_crawler.ts
@@ -47,7 +47,7 @@ const crawler = new PlaywrightCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/puppeteer_crawler.ts b/docs/examples/puppeteer_crawler.ts
index 6172ea0f2..1f365f5c7 100644
--- a/docs/examples/puppeteer_crawler.ts
+++ b/docs/examples/puppeteer_crawler.ts
@@ -47,7 +47,7 @@ const crawler = new PuppeteerCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdx
index 520dcc62f..c987dc732 100644
--- a/docs/introduction/03-adding-urls.mdx
+++ b/docs/introduction/03-adding-urls.mdx
@@ -43,7 +43,7 @@ The process is simple:
4. Visit the newly enqueued links.
5. Repeat the process.
-In the following paragraphs you will learn about the <ApiLink to="core/function/enqueueLinks">`enqueueLinks`</ApiLink> function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
+In the following paragraphs you will learn about the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks`</ApiLink> function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
:::tip `enqueueLinks` context awareness
@@ -130,7 +130,7 @@ await enqueueLinks({
### Filter URLs with patterns
-For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. <ApiLink to="core/interface/EnqueueLinksOptions">See the reference</ApiLink> for more information about them as well as other options.
+For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. <ApiLink to="core/interface/EnqueueUrlsOptions">See the reference</ApiLink> for more information about them as well as other options.
:::caution Defaults override
@@ -146,7 +146,7 @@ await enqueueLinks({
### Transform requests
-To have absolute control, we have the <ApiLink to="core/interface/EnqueueLinksOptions/#transformRequestFunction">`transformRequestFunction`</ApiLink>. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+To have absolute control, we have the <ApiLink to="core/interface/EnqueueUrlsOptions#transformRequestFunction">`transformRequestFunction`</ApiLink>. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
```ts
await enqueueLinks({
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
index 26f65da7a..c709b6431 100644
--- a/docs/public-api/crawlee-basic.api.md
+++ b/docs/public-api/crawlee-basic.api.md
@@ -21,6 +21,7 @@ import { Dataset } from '@crawlee/core';
import type { DatasetExportOptions } from '@crawlee/core';
import type { Dictionary } from '@crawlee/types';
import { Element as Element_2 } from '@crawlee/utils/internal';
+import type { EnqueueUrlsOptions } from '@crawlee/core';
import { EventManager } from '@crawlee/core';
import type { FinalStatistics } from '@crawlee/core';
import type { GetUserDataFromRequest } from '@crawlee/core';
@@ -282,7 +283,7 @@ export { CheerioAPI }
export { CheerioRoot }
// @public (undocumented)
-export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions {
+export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions, EnqueueUrlsOptions {
}
// @public (undocumented)
diff --git a/docs/public-api/crawlee-browser.api.md b/docs/public-api/crawlee-browser.api.md
index 20d5b581a..c9634ef49 100644
--- a/docs/public-api/crawlee-browser.api.md
+++ b/docs/public-api/crawlee-browser.api.md
@@ -4,11 +4,11 @@
```ts
+import type { AddRequestsBatchedResult } from '@crawlee/basic';
import type { Awaitable } from '@crawlee/types';
import { BaseHttpClient } from '@crawlee/http-client';
import { BasicCrawler } from '@crawlee/basic';
import { BasicCrawlerOptions } from '@crawlee/basic';
-import type { BatchAddRequestsResult } from '@crawlee/types';
import type { BrowserController } from '@crawlee/browser-pool';
import type { BrowserPlugin } from '@crawlee/browser-pool';
import type { BrowserPluginOptions } from '@crawlee/browser-pool';
@@ -23,6 +23,7 @@ import type { Dictionary } from '@crawlee/types';
import type { EnqueueLinksOptions } from '@crawlee/basic';
import type { ErrorHandler } from '@crawlee/basic';
import { EventManager } from '@crawlee/basic';
+import type { ExtractLinksOptions } from '@crawlee/basic';
import type { GetUserDataFromRequest } from '@crawlee/basic';
import type { IBrowserPool } from '@crawlee/types';
import type { InferBrowserPluginArray } from '@crawlee/browser-pool';
@@ -196,7 +197,8 @@ export interface BrowserCrawlerOptions<Page extends CommonPage = CommonPage, Res
// @public (undocumented)
export interface BrowserCrawlingContext<Page extends CommonPage = CommonPage, Response extends BaseResponse = BaseResponse, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
GoToOptions extends Dictionary = Dictionary> extends CrawlingContext<UserData> {
- enqueueLinks: (options?: EnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ enqueueLinks: (options?: EnqueueLinksOptions) => Promise<AddRequestsBatchedResult>;
+ extractLinks: (options?: ExtractLinksOptions) => Promise<string[]>;
gotoOptions: GoToOptions;
page: Page;
request: LoadedRequest<Request_2<UserData>>;
diff --git a/docs/public-api/crawlee-cheerio.api.md b/docs/public-api/crawlee-cheerio.api.md
index f61a8121d..d2740d96c 100644
--- a/docs/public-api/crawlee-cheerio.api.md
+++ b/docs/public-api/crawlee-cheerio.api.md
@@ -4,7 +4,7 @@
```ts
-import type { BatchAddRequestsResult } from '@crawlee/types';
+import type { AddRequestsBatchedResult } from '@crawlee/http';
import * as cheerio from 'cheerio';
import type { CheerioAPI } from 'cheerio';
import { CheerioRoot } from '@crawlee/utils/internal';
@@ -13,6 +13,7 @@ import type { CrawlingContext } from '@crawlee/http';
import type { Dictionary } from '@crawlee/types';
import type { EnqueueLinksOptions } from '@crawlee/http';
import type { ErrorHandler } from '@crawlee/http';
+import type { ExtractLinksOptions } from '@crawlee/http';
import type { GetUserDataFromRequest } from '@crawlee/http';
import { HttpCrawler } from '@crawlee/http';
import type { HttpCrawlerOptions } from '@crawlee/http';
@@ -32,7 +33,8 @@ export class CheerioCrawler<ContextExtension = Dictionary<never>, ExtendedContex
readonly body: string;
readonly $: CheerioAPI;
} & {
- enqueueLinks: (enqueueOptions?: EnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ extractLinks: (options?: ExtractLinksOptions) => Promise<string[]>;
+ enqueueLinks: (options?: EnqueueLinksOptions) => Promise<AddRequestsBatchedResult>;
waitForSelector: (selector: string, _timeoutMs?: number) => Promise<void>;
parseWithCheerio: (selector?: string, timeoutMs?: number) => Promise<CheerioAPI>;
}>;
@@ -49,7 +51,8 @@ export interface CheerioCrawlingContext<UserData extends Dictionary = any, // wi
JSONData extends Dictionary = any> extends InternalHttpCrawlingContext<UserData, JSONData> {
$: cheerio.CheerioAPI;
body: string;
- enqueueLinks(options?: EnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
waitForSelector(selector: string, timeoutMs?: number): Promise<void>;
}
diff --git a/docs/public-api/crawlee-core.api.md b/docs/public-api/crawlee-core.api.md
index 681c17f93..4b54d2473 100644
--- a/docs/public-api/crawlee-core.api.md
+++ b/docs/public-api/crawlee-core.api.md
@@ -38,12 +38,10 @@ import type { QueueOperationInfo } from '@crawlee/types';
import type { ReadonlyDeep } from 'type-fest';
import type { RequestQueueBackend } from '@crawlee/types';
import type { RequestQueueInfo } from '@crawlee/types';
-import { RobotsTxtFile } from '@crawlee/utils';
import { schemas } from '@crawlee/utils/internal';
import type { SendRequestOptions } from '@crawlee/types';
import type { SessionFingerprint } from '@crawlee/types';
import { SessionState } from '@crawlee/types';
-import type { SetRequired } from 'type-fest';
import type { StandardSchemaV1 } from '@standard-schema/spec';
import type * as storage from '@crawlee/types';
import { StorageBackend } from '@crawlee/types';
@@ -333,7 +331,6 @@ export { CrawleeLoggerOptions }
// @public (undocumented)
export interface CrawlingContext<UserData extends Dictionary = Dictionary> extends RestrictedCrawlingContext<UserData> {
- enqueueLinks(options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>> & Pick<EnqueueLinksOptions, 'requestManager' | 'robotsTxtFile'>): Promise<unknown>;
extendTimeout(secs: number): void;
registerDeferredCleanup(cleanup: () => Promise<unknown>): void;
sendRequest: (requestOverrides?: Partial<HttpRequestOptions>, optionsOverrides?: SendRequestOptions) => Promise<Response>;
@@ -529,43 +526,35 @@ export interface DefaultStorageIdentifier {
}
// @public
-export function enqueueLinks(options: SetRequired<Omit<EnqueueLinksOptions, 'requestManager'>, 'urls'> & {
- requestManager: {
- addRequestsBatched: (requests: Request_2<Dictionary>[], options: AddRequestsBatchedOptions) => Promise<AddRequestsBatchedResult>;
- };
-}): Promise<BatchAddRequestsResult>;
+export type EnqueueLinksOptions = ExtractLinksOptions & EnqueueUrlsOptions;
-// @public (undocumented)
-export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
+// @public
+export enum EnqueueStrategy {
+ All = "all",
+ SameDomain = "same-domain",
+ SameHostname = "same-hostname",
+ SameOrigin = "same-origin"
+}
+
+// @public
+export type EnqueueStrategyOption = EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
+
+// @public
+export interface EnqueueUrlsOptions extends RequestQueueOperationOptions {
baseUrl?: string;
exclude?: readonly UrlPatternInput[];
include?: readonly UrlPatternInput[];
label?: string;
limit?: number;
onSkippedRequest?: SkippedRequestCallback;
- requestManager?: IRequestManager;
- respectRobotsTxtFile?: boolean | {
- userAgent?: string;
- };
- robotsTxtFile?: Pick<RobotsTxtFile, 'isAllowed'>;
- selector?: string;
sessionId?: string;
skipNavigation?: boolean;
- strategy?: EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
+ strategy?: EnqueueStrategyOption;
transformRequestFunction?: RequestTransform;
- urls?: readonly string[];
userData?: Dictionary;
waitForAllRequestsToBeAdded?: boolean;
}
-// @public
-export enum EnqueueStrategy {
- All = "all",
- SameDomain = "same-domain",
- SameHostname = "same-hostname",
- SameOrigin = "same-origin"
-}
-
// @public
export interface ErrnoException extends Error {
// (undocumented)
@@ -743,6 +732,12 @@ export type ExplicitStorageIdentifier = {
alias: string;
};
+// @public
+export interface ExtractLinksOptions {
+ baseUrl?: string;
+ selector?: string;
+}
+
// @public (undocumented)
export function field<T extends z.ZodType>(schema: T, envVar?: string | string[]): ConfigField<T>;
@@ -1584,8 +1579,7 @@ export interface ResponseLike {
// @public (undocumented)
export interface RestrictedCrawlingContext<UserData extends Dictionary = Dictionary> {
- addRequests: (requestsLike: ReadonlyDeep<(string | Source)[]>, options?: ReadonlyDeep<RequestQueueOperationOptions>) => Promise<void>;
- enqueueLinks: (options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>>) => Promise<unknown>;
+ addRequests: (requestsLike: ReadonlyDeep<(string | Source)[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
getKeyValueStore: (identifier?: string | StorageIdentifier) => Promise<Pick<KeyValueStore, 'id' | 'name' | 'getValue' | 'getAutoSavedValue' | 'setValue' | 'getPublicUrl'>>;
// (undocumented)
id: string;
@@ -1609,15 +1603,15 @@ export interface RouteOptions {
}
// @public
-export class Router<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'>, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
+export class Router<Context extends RestrictedCrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> {
addDefaultHandler<UserData extends Dictionary = DefaultRouteUserData<Routes, GetUserDataFromRequest<Context['request']>>>(handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
addHandler<Label extends keyof Routes & string>(label: Label, handler: (ctx: RouterHandlerContext<Context, Routes[Label], Routes>) => Awaitable<void>, options?: RouteOptions): void;
addHandler<UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(label: RouterLabel<Routes>, handler: (ctx: RouterHandlerContext<Context, UserData, Routes>) => Awaitable<void>, options?: RouteOptions): void;
- static create<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
// (undocumented)
- static create<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, UserData extends Dictionary = GetUserDataFromRequest<Context['request']>>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
// (undocumented)
- static create<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext, const Schemas extends RouteSchemas = RouteSchemas>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
getHandler(label?: string | symbol): (ctx: Context) => Awaitable<void>;
getMaxTimeoutSecs(): number | undefined;
getTimeoutSecs(label?: string | symbol): number | undefined;
@@ -1625,7 +1619,7 @@ export class Router<Context extends Omit<RestrictedCrawlingContext, 'enqueueLink
}
// @public (undocumented)
-export interface RouterHandler<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends Router<Context, Routes> {
+export interface RouterHandler<Context extends RestrictedCrawlingContext = CrawlingContext, Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>> extends Router<Context, Routes> {
// (undocumented)
(ctx: Context): Awaitable<void>;
}
@@ -2214,7 +2208,7 @@ export interface ThrottlingRequestManagerOptions<T extends IRequestManager = IRe
export { tryAbsoluteURL }
// @public
-export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>, options?: ReadonlyDeep<RequestQueueOperationOptions>) => Promise<void>;
+export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>, options?: ReadonlyDeep<EnqueueUrlsOptions>) => Promise<AddRequestsBatchedResult>;
// @public
export type TypedContextEnqueueLinks<EnqueueLinks, Routes extends Record<keyof Routes, Dictionary>> = EnqueueLinks extends (options?: infer Options) => infer Result ? (options?: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks extends (options: infer Options) => infer Result ? (options: TypedEnqueueLinksOptions<Options, Routes>) => Result : EnqueueLinks;
diff --git a/docs/public-api/crawlee-jsdom.api.md b/docs/public-api/crawlee-jsdom.api.md
index 50d2ab91a..71a0e4cde 100644
--- a/docs/public-api/crawlee-jsdom.api.md
+++ b/docs/public-api/crawlee-jsdom.api.md
@@ -4,8 +4,8 @@
```ts
+import type { AddRequestsBatchedResult } from '@crawlee/http';
import { BaseHttpClient } from '@crawlee/http-client';
-import type { BatchAddRequestsResult } from '@crawlee/types';
import { CheerioAPI } from 'cheerio';
import { CheerioRoot } from '@crawlee/utils/internal';
import { Configuration } from '@crawlee/http';
@@ -16,6 +16,7 @@ import type { DOMWindow } from 'jsdom';
import type { EnqueueLinksOptions } from '@crawlee/http';
import type { ErrorHandler } from '@crawlee/http';
import { EventManager } from '@crawlee/http';
+import type { ExtractLinksOptions } from '@crawlee/http';
import type { GetUserDataFromRequest } from '@crawlee/http';
import { HttpCrawler } from '@crawlee/http';
import type { HttpCrawlerOptions } from '@crawlee/http';
@@ -47,7 +48,8 @@ export class JSDOMCrawler<ContextExtension = Dictionary<never>, ExtendedContext
readonly body: string;
readonly document: Document;
} & {
- enqueueLinks: (enqueueOptions?: EnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ extractLinks: (options?: ExtractLinksOptions) => Promise<string[]>;
+ enqueueLinks: (options?: EnqueueLinksOptions) => Promise<AddRequestsBatchedResult>;
waitForSelector(selector: string, timeoutMs?: number): Promise<void>;
parseWithCheerio(selector?: string, _timeoutMs?: number): Promise<CheerioAPI>;
}>;
@@ -179,7 +181,8 @@ JSONData extends Dictionary = any> extends InternalHttpCrawlingContext<UserData,
body: string;
// (undocumented)
document: Document;
- enqueueLinks(options?: EnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
waitForSelector(selector: string, timeoutMs?: number): Promise<void>;
// (undocumented)
diff --git a/docs/public-api/crawlee-linkedom.api.md b/docs/public-api/crawlee-linkedom.api.md
index a88c86bc1..44a404fac 100644
--- a/docs/public-api/crawlee-linkedom.api.md
+++ b/docs/public-api/crawlee-linkedom.api.md
@@ -4,7 +4,7 @@
```ts
-import type { BatchAddRequestsResult } from '@crawlee/types';
+import type { AddRequestsBatchedResult } from '@crawlee/http';
import * as cheerio from 'cheerio';
import { CheerioRoot } from '@crawlee/utils/internal';
import { ContextPipeline } from '@crawlee/http';
@@ -12,6 +12,7 @@ import type { CrawlingContext } from '@crawlee/http';
import type { Dictionary } from '@crawlee/types';
import type { EnqueueLinksOptions } from '@crawlee/http';
import type { ErrorHandler } from '@crawlee/http';
+import type { ExtractLinksOptions } from '@crawlee/http';
import type { GetUserDataFromRequest } from '@crawlee/http';
import { HttpCrawler } from '@crawlee/http';
import type { HttpCrawlerOptions } from '@crawlee/http';
@@ -41,16 +42,13 @@ export class LinkeDOMCrawler<ContextExtension = Dictionary<never>, ExtendedConte
readonly body: string;
readonly document: Document;
} & {
- enqueueLinks: (enqueueOptions?: LinkeDOMCrawlerEnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ extractLinks: (options?: ExtractLinksOptions) => Promise<string[]>;
+ enqueueLinks: (options?: EnqueueLinksOptions) => Promise<AddRequestsBatchedResult>;
waitForSelector(selector: string, timeoutMs?: number): Promise<void>;
parseWithCheerio(selector?: string, _timeoutMs?: number): Promise<cheerio.CheerioAPI>;
}>;
}
-// @public (undocumented)
-export interface LinkeDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestManager'> {
-}
-
// @public (undocumented)
export interface LinkeDOMCrawlerOptions<ContextExtension = Dictionary<never>, ExtendedContext extends LinkeDOMCrawlingContext = LinkeDOMCrawlingContext & ContextExtension, UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -62,7 +60,8 @@ export interface LinkeDOMCrawlingContext<UserData extends Dictionary = any, // w
JSONData extends Dictionary = any> extends InternalHttpCrawlingContext<UserData, JSONData> {
// (undocumented)
document: Document;
- enqueueLinks(options?: LinkeDOMCrawlerEnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
waitForSelector(selector: string, timeoutMs?: number): Promise<void>;
// (undocumented)
diff --git a/docs/public-api/crawlee-playwright.api.md b/docs/public-api/crawlee-playwright.api.md
index d309831d8..f43ea9079 100644
--- a/docs/public-api/crawlee-playwright.api.md
+++ b/docs/public-api/crawlee-playwright.api.md
@@ -82,6 +82,7 @@ export class AdaptivePlaywrightCrawler<ContextExtension = Dictionary<never>, Ext
readonly querySelectorAll: AdaptivePlaywrightCrawlerContext["querySelectorAll"];
readonly waitForSelector: AdaptivePlaywrightCrawlerContext["waitForSelector"];
readonly parseWithCheerio: AdaptivePlaywrightCrawlerContext["parseWithCheerio"];
+ readonly enqueueLinks: AdaptivePlaywrightCrawlerContext["enqueueLinks"];
}>;
// (undocumented)
protected init(): Promise<void>;
diff --git a/docs/public-api/crawlee.api.md b/docs/public-api/crawlee.api.md
index 87b1b818c..86806b621 100644
--- a/docs/public-api/crawlee.api.md
+++ b/docs/public-api/crawlee.api.md
@@ -5,7 +5,6 @@
```ts
import { downloadListOfUrls } from '@crawlee/utils';
-import { enqueueLinks } from '@crawlee/core';
import { Log } from '@apify/log';
import { parseOpenGraph } from '@crawlee/utils';
import { playwrightUtils } from '@crawlee/playwright';
@@ -18,7 +17,6 @@ export const utils: {
puppeteer: typeof puppeteerUtils;
playwright: typeof playwrightUtils;
log: Log;
- enqueueLinks: typeof enqueueLinks;
social: typeof social;
sleep: typeof sleep;
downloadListOfUrls: typeof downloadListOfUrls;
diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.md
index ce14f1e3e..7af8ef332 100644
--- a/docs/upgrading/upgrading_v4.md
+++ b/docs/upgrading/upgrading_v4.md
@@ -51,6 +51,7 @@ The purely mechanical renames, collected in one place. Where a row links to a se
| crawler options `requestList` / `requestQueue` | `requestManager` ([details](#crawler-requestlist--requestqueue-options-deprecated-in-favor-of-requestmanager)) |
| `enqueueLinks({ requestQueue })` | `enqueueLinks({ requestManager })` |
| `enqueueLinks({ globs, regexps, pseudoUrls })` | `enqueueLinks({ include })` ([details](#globs-regexps-and-pseudourls-replaced-by-include)) |
+| `(await enqueueLinks()).processedRequests` | `(await enqueueLinks()).addedRequests` ([details](#enqueuelinks-return-value-reshaped-addrequestsbatchedresult-instead-of-batchaddrequestsresult)) |
| `autoscaledPoolOptions` | `taskLoopOptions` ([narrowed](#autoscaledpooloptions-is-now-taskloopoptions-and-no-longer-carries-concurrency-config)) |
| `gotScraping` (from `@crawlee/utils`) | `GotScrapingHttpClient` (`@crawlee/got-scraping-client`) |
| `SDK_`-prefixed internal KVS keys | `CRAWLEE_`-prefixed ([details](#internal-kvs-keys-renamed)) |
@@ -737,6 +738,47 @@ The `transformRequestFunction` callback receives a `RequestOptions` object and c
- `'unchanged'` to keep the original options as-is
- A falsy value or `'skip'` to exclude the request from the queue
+### `enqueueLinks()` return value reshaped: `AddRequestsBatchedResult` instead of `BatchAddRequestsResult`
+
+`enqueueLinks()` (and `context.addRequests()`) now return the same `AddRequestsBatchedResult` object that `crawler.addRequests()` / `queue.addRequestsBatched()` already returned in v3, instead of repackaging it into the legacy `BatchAddRequestsResult` shape:
+
+| Before (`BatchAddRequestsResult`) | After (`AddRequestsBatchedResult`) |
+| --- | --- |
+| `processedRequests` | `addedRequests` |
+| `unprocessedRequests` (always `[]` — `enqueueLinks()` never actually populated it) | *(removed — retries are handled internally, see "`RequestQueue.addRequestsBatched` no longer retries rejected requests" below)* |
+| *(not exposed)* | `waitForAllRequestsToBeAdded` — a promise resolving with the requests added in batches after the first (new) |
+| *(not exposed)* | `requestsOverLimit` — requests dropped because of the `limit` / `maxRequestsPerCrawl` budget (new) |
+
+**Before:**
+```typescript
+const { processedRequests } = await enqueueLinks();
+```
+
+**After:**
+```typescript
+const { addedRequests } = await enqueueLinks();
+```
+
+### `extractLinks()`: extracting URLs without enqueueing them
+
+Crawling contexts that support `enqueueLinks()` (Cheerio, JSDOM, LinkeDOM, and browser-based crawlers) now also expose an `extractLinks()` helper that returns the matching URLs as strings, without adding them to the request queue:
+
+```typescript
+const urls = await extractLinks({ selector: '.product-link' });
+```
+
+`enqueueLinks()` itself is unchanged in behavior — it now calls `extractLinks()` internally and forwards the URLs to `context.addRequests()`.
+
+### `context.addRequests()` now applies `enqueueLinks`-style filtering, and `BasicCrawler` no longer has `enqueueLinks`
+
+`context.addRequests()` (and `crawler.addRequests()`) accept the same `include` / `exclude` / `strategy` / `transformRequestFunction` / `onSkippedRequest` options as `enqueueLinks()`, and resolve `baseUrl`-relative URLs the same way. Unlike `enqueueLinks()`, there is no implicit "current page" to anchor the strategy to, so `strategy` defaults to `EnqueueStrategy.All` here instead of `EnqueueStrategy.SameHostname`.
+
+`context.addRequests()` also now returns an `AddRequestsBatchedResult` (previously it resolved to `void`).
+
+`BasicCrawler` (and its `BasicCrawlingContext`) no longer has an `enqueueLinks()` method — `BasicCrawler` has no concept of a page to extract links from. `enqueueLinks()` remains available on crawlers with web content (`CheerioCrawler`, `HttpCrawler`-derived crawlers, `PlaywrightCrawler`, `PuppeteerCrawler`, etc.), now implemented in terms of `extractLinks()` + `addRequests()`.
+
+The `robotsTxtFile` / `respectRobotsTxtFile` per-call options are removed from `enqueueLinks()` — robots.txt filtering is applied by the crawler consistently via `BasicCrawlerOptions.respectRobotsTxtFile`.
+
### Internal KVS keys renamed
Several internal Crawlee keys were prefixed with the `SDK_` prefix for legacy reasons — these keys now start with `CRAWLEE_` instead. These are, e.g., `CRAWLEE_SESSION_POOL_STATE` or `CRAWLEE_CRAWLER_STATISTICS_{n}`.
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index b9b991d64..b465bcf85 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -9,7 +9,7 @@ import type {
CrawleeLogger,
CrawlingContext,
DatasetExportOptions,
- EnqueueLinksOptions,
+ EnqueueUrlsOptions,
EventStatusMessageData,
FinalStatistics,
GetUserDataFromRequest,
@@ -18,37 +18,42 @@ import type {
IRequestLoader,
IRequestManager,
IStatistics,
- Request,
RequestsLike,
- RequestTransform,
RouterHandler,
RouterRoutes,
SkippedRequestCallback,
+ SkippedRequestReason,
Source,
StatisticState,
StorageIdentifier,
StorageWritePolicy,
TaskLoopPredicates,
TypedRequestsLike,
+ UrlPatternObject,
} from '@crawlee/core';
import {
+ applyRequestTransform,
AutoscaledPool,
bindMethodsToServiceLocator,
BLOCKED_STATUS_CODES,
+ buildEnqueueStrategyPatterns,
ConcurrencySystem,
Configuration,
+ constructUrlPatternObjects,
ContextPipeline,
ContextPipelineCleanupError,
ContextPipelineInitializationError,
ContextPipelineInterruptedError,
+ createRequestOptions,
createStorageTransaction,
+ Request,
CriticalError,
currentStorageTransaction,
Dataset,
- enqueueLinks,
EnqueueStrategy,
EventManager,
EventType,
+ filterRequestOptionsByPatterns,
getObjectType,
KeyValueStore,
log,
@@ -83,7 +88,6 @@ import {
import { BaseHttpClient, FetchHttpClient } from '@crawlee/http-client';
import type {
Awaitable,
- BatchAddRequestsResult,
Dictionary,
ISession,
ISessionPool,
@@ -94,7 +98,7 @@ import type {
import { isAsyncIterable, isIterable, ROTATE_PROXY_ERRORS } from '@crawlee/utils/internal';
import { RobotsTxtFile } from '@crawlee/utils';
import { getDomain } from 'tldts';
-import type { ReadonlyDeep, SetRequired } from 'type-fest';
+import type { ReadonlyDeep } from 'type-fest';
import { z } from 'zod';
import { LruCache } from '@apify/datastructures';
@@ -153,6 +157,35 @@ const deferredCleanupKey = Symbol('deferredCleanup');
// The request timeout plumbing (the window helper, the context symbols, and the race) lives in its own module.
export { navigationDeadlineKey, remainingNavigationWindowMillis } from './request-timeout.js';
+const urlPatternSchema = z.union([
+ z.string(),
+ z.instanceof(RegExp),
+ schemas.objectWithKeys(['glob']),
+ schemas.objectWithKeys(['regexp']),
+]);
+
+// `looseObject` (rather than `strictObject`) lets subclasses forward their own extraction-only options
+// (e.g. `selector`) straight through without having to strip them out first.
+const addRequestsOptionsSchema = z.looseObject({
+ forefront: z.boolean().optional(),
+ cache: z.boolean().optional(),
+ waitForAllRequestsToBeAdded: z.boolean().optional(),
+ batchSize: schemas.anyNumber.optional(),
+ waitBetweenBatchesMillis: schemas.anyNumber.optional(),
+ maxNewRequests: schemas.anyNumber.optional(),
+ limit: schemas.anyNumber.optional(),
+ baseUrl: z.string().optional(),
+ userData: schemas.anyObject.optional(),
+ label: z.string().optional(),
+ sessionId: z.string().optional(),
+ skipNavigation: z.boolean().optional(),
+ include: schemas.arrayOf(urlPatternSchema, 'URL patterns').min(1).optional(),
+ exclude: schemas.arrayOf(urlPatternSchema, 'URL patterns').optional(),
+ transformRequestFunction: schemas.anyFunction.optional(),
+ strategy: z.enum(EnqueueStrategy).optional(),
+ onSkippedRequest: schemas.anyFunction.optional(),
+});
+
/** The in-flight context, carrying the timeout slots ({@apilink raceWithTimeout} hangs its extender on them). */
type PendingCrawlingContext = { request: Request } & Partial<CrawlingContext> & RequestTimeoutContext;
@@ -462,9 +495,6 @@ export interface BasicCrawlerOptions<
* 2. because they don't match enqueueLinks filters,
* 3. because they are redirected to a URL that doesn't match the enqueueLinks strategy,
* 4. or because the {@apilink BasicCrawlerOptions.maxRequestsPerCrawl|`maxRequestsPerCrawl`} limit has been reached
- *
- * When `enqueueLinks` is called with its own `onSkippedRequest` callback, both are invoked — this one first,
- * then the `enqueueLinks` one.
*/
onSkippedRequest?: SkippedRequestCallback;
@@ -828,7 +858,7 @@ export class BasicCrawler<
/** The resolved per-storage-type write policy overrides forwarded to each request's transaction. */
readonly #storageWritePolicy: Partial<StorageWritePolicy>;
protected readonly onSkippedRequest?: SkippedRequestCallback;
- #ownsEventManager = false;
+ #closeEvents?: boolean;
#loggedPerRun = new Set<string>();
readonly #robotsTxtFileCache: LruCache<RobotsTxtFile>;
protected readonly identity: CrawlerIdentity;
@@ -1414,21 +1444,16 @@ export class BasicCrawler<
}
private async createContextHelpers({ request, session }: { request: Request; session: ISession }) {
- const enqueueLinksWrapper: CrawlingContext['enqueueLinks'] = async (options) => {
- const requestManager = await this.getRequestManager();
-
- return await this.enqueueLinksWithCrawlDepth(options, request!, requestManager);
- };
const addRequests: CrawlingContext['addRequests'] = async (requests, options = {}) => {
const newCrawlDepth = request!.crawlDepth + 1;
const requestsGenerator = this.addCrawlDepthRequestGenerator(requests, newCrawlDepth);
- await this.addRequests(requestsGenerator, options);
+ return await this.addRequests(requestsGenerator, options);
};
const sendRequest = createSendRequest(this.httpClient, request!, session);
- return { enqueueLinks: enqueueLinksWrapper, addRequests, sendRequest };
+ return { addRequests, sendRequest };
}
private buildFinalContextPipeline(): ContextPipeline<CrawlingContext, ExtendedContext> {
@@ -1661,8 +1686,8 @@ export class BasicCrawler<
try {
await this.#autoscaledPool!.run();
} finally {
- await this.stats.stopCapturing();
await this.teardown();
+ await this.stats.stopCapturing();
process.off('SIGINT', sigintHandler);
eventManager.off(EventType.MIGRATING, boundPauseOnMigration);
@@ -1929,6 +1954,11 @@ export class BasicCrawler<
* the batches via `waitBetweenBatchesMillis`. If you want to wait for all batches to be added to the queue, you can use
* the `waitForAllRequestsToBeAdded` promise you get in the response object.
*
+ * Optionally, the requests can be filtered using `include`/`exclude` glob or regexp patterns and an
+ * enqueue `strategy` (both AND-ed together, same as {@apilink CrawlingContext.enqueueLinks|`enqueueLinks`}),
+ * relative to `baseUrl`. Unlike `enqueueLinks`, there is no implicit "current page" to anchor the strategy
+ * to, so `strategy` defaults to {@apilink EnqueueStrategy.All|`all`} here.
+ *
* This is an alias for calling `addRequestsBatched()` on the implicit `RequestQueue` for this crawler instance.
*
* @param requests The requests to add
@@ -1940,70 +1970,139 @@ export class BasicCrawler<
): Promise<CrawlerAddRequestsResult> {
await this.getRequestManager();
- const requestLimit = await this.calculateEnqueuedRequestLimit();
+ if (!isIterable(requests) && !isAsyncIterable(requests)) {
+ throw new Error(`Expected an iterable or async iterable, got ${getObjectType(requests)}`);
+ }
- const skippedBecauseOfRobots = new Set<string>();
- const skippedBecauseOfMaxCrawlDepth = new Set<string>();
+ parseArgument(options, addRequestsOptionsSchema, 'EnqueueUrlsOptions');
+
+ // `label`/`userData` apply to every request this call produces, so a single upfront validation
+ // against the label's schema covers them all and fails the whole call fast, rather than failing
+ // lazily once the generator below is drained. Skipped when neither is set - each item still gets
+ // its own per-item validation below, and validating an absent label/userData here would spuriously
+ // check them against a registered default-route schema.
+ if (options.label !== undefined || options.userData !== undefined) {
+ await this.validateRequestUserData({ label: options.label, userData: options.userData });
+ }
+
+ const requestLimit = await this.calculateEnqueuedRequestLimit(options.limit);
+
+ const strategy = options.strategy ?? EnqueueStrategy.All;
+ const urlExcludePatternObjects: UrlPatternObject[] = options.exclude?.length
+ ? constructUrlPatternObjects(options.exclude)
+ : [];
+ const urlPatternObjects: UrlPatternObject[] = options.include?.length
+ ? constructUrlPatternObjects(options.include)
+ : [];
+ // The strategy always applies, even when `include` patterns are provided - the two are AND-ed together
+ // (a URL must match an `include` pattern *and* satisfy the strategy). This mirrors crawlee-python.
+ const enqueueStrategyPatterns: UrlPatternObject[] = options.baseUrl
+ ? buildEnqueueStrategyPatterns(options.baseUrl, strategy)
+ : [];
const isAllowedBasedOnRobotsTxtFile = this.isAllowedBasedOnRobotsTxtFile.bind(this);
const maxCrawlDepth = this.maxCrawlDepth;
const validateRequestUserData = this.validateRequestUserData.bind(this);
- if (!isIterable(requests) && !isAsyncIterable(requests)) {
- throw new Error(`Expected an iterable or async iterable, got ${getObjectType(requests)}`);
- }
+ const allSkipped: { url: string; reason: SkippedRequestReason }[] = [];
async function* filteredRequests() {
for await (const request of requests) {
- const url = typeof request === 'string' ? request : request.url!;
+ const [requestOptions] = createRequestOptions(
+ [typeof request === 'string' ? request : (request as Record<string, unknown>)],
+ { ...options, strategy },
+ );
- if (maxCrawlDepth !== undefined && (request as any).crawlDepth > maxCrawlDepth) {
- skippedBecauseOfMaxCrawlDepth.add(url);
+ if (!requestOptions) {
+ continue; // invalid URL, silently dropped (matches `createRequestOptions`'s own filtering)
+ }
+
+ if (maxCrawlDepth !== undefined && requestOptions.crawlDepth! > maxCrawlDepth) {
+ allSkipped.push({ url: requestOptions.url, reason: 'depth' });
continue;
}
- if (await isAllowedBasedOnRobotsTxtFile(url)) {
- await validateRequestUserData(request);
- yield request;
- } else {
- skippedBecauseOfRobots.add(url);
+ if (!(await isAllowedBasedOnRobotsTxtFile(requestOptions.url))) {
+ allSkipped.push({ url: requestOptions.url, reason: 'robotsTxt' });
+ continue;
+ }
+
+ const onSkippedFilterUrl = (url: string) => allSkipped.push({ url, reason: 'filters' });
+
+ // Filter by user patterns first (with exclude)...
+ let filtered = filterRequestOptionsByPatterns(
+ [requestOptions],
+ urlPatternObjects.length > 0 ? urlPatternObjects : undefined,
+ urlExcludePatternObjects,
+ strategy,
+ onSkippedFilterUrl,
+ );
+ // ...then filter by the enqueue strategy (making this an AND check)
+ filtered = filterRequestOptionsByPatterns(
+ filtered,
+ enqueueStrategyPatterns.length > 0 ? enqueueStrategyPatterns : undefined,
+ [],
+ strategy,
+ onSkippedFilterUrl,
+ );
+
+ if (filtered.length === 0) {
+ continue;
}
+
+ let [finalOptions] = filtered;
+
+ if (options.transformRequestFunction) {
+ const transformed = applyRequestTransform([finalOptions], options.transformRequestFunction, (r) =>
+ allSkipped.push({ url: r.url, reason: r.skippedReason ?? 'transform' }),
+ );
+
+ if (transformed.length === 0) {
+ continue;
+ }
+
+ [finalOptions] = transformed;
+ }
+
+ await validateRequestUserData(finalOptions);
+ yield new Request(finalOptions);
}
}
const result = await this.requestManager!.addRequestsBatched(filteredRequests(), {
- ...options,
+ forefront: options.forefront,
+ waitForAllRequestsToBeAdded: options.waitForAllRequestsToBeAdded,
+ batchSize: options.batchSize,
+ waitBetweenBatchesMillis: options.waitBetweenBatchesMillis,
maxNewRequests: requestLimit,
});
- // Report requests skipped due to the maxNewRequests budget (i.e. maxRequestsPerCrawl limit)
- const skippedBecauseOfLimit = result.requestsOverLimit ?? [];
-
- if (skippedBecauseOfRobots.size > 0) {
- this.log.warning(`Some requests were skipped because they were disallowed based on the robots.txt file`, {
- skipped: [...skippedBecauseOfRobots],
- });
+ // Report requests skipped due to the maxNewRequests budget (i.e. maxRequestsPerCrawl limit, or an
+ // explicit `limit` option)
+ for (const request of result.requestsOverLimit ?? []) {
+ allSkipped.push({ url: typeof request === 'string' ? request : request.url!, reason: 'limit' });
}
- if (
- skippedBecauseOfRobots.size > 0 ||
- skippedBecauseOfLimit.length > 0 ||
- skippedBecauseOfMaxCrawlDepth.size > 0
- ) {
+ if (allSkipped.length > 0) {
+ const skippedRobotsUrls = allSkipped.filter((s) => s.reason === 'robotsTxt').map((s) => s.url);
+ if (skippedRobotsUrls.length > 0) {
+ this.log.warning(
+ `Some requests were skipped because they were disallowed based on the robots.txt file`,
+ { skipped: skippedRobotsUrls },
+ );
+ }
+
+ // Only log the limit message when an explicit `limit` was passed (not the internal
+ // `maxRequestsPerCrawl`-derived one), and only once per call.
+ if (options.limit !== undefined && allSkipped.some((s) => s.reason === 'limit')) {
+ this.log.info(`Skipping requests in this call due to the enqueueLinks limit of ${options.limit}.`);
+ }
+
await Promise.all(
- [...skippedBecauseOfRobots]
- .map((url) => {
- return this.handleSkippedRequest({ url, reason: 'robotsTxt' });
- })
- .concat(
- skippedBecauseOfLimit.map((request) => {
- const url = typeof request === 'string' ? request : request.url!;
- return this.handleSkippedRequest({ url, reason: 'limit' });
- }),
- [...skippedBecauseOfMaxCrawlDepth].map((url) => {
- return this.handleSkippedRequest({ url, reason: 'depth' });
- }),
- ),
+ allSkipped.map(async ({ url, reason }) => {
+ await this.handleSkippedRequest({ url, reason });
+ await options.onSkippedRequest?.({ url, reason });
+ }),
);
}
@@ -2104,7 +2203,7 @@ export class BasicCrawler<
if (!eventManager.isInitialized()) {
await eventManager.init();
- this.#ownsEventManager = true;
+ this.#closeEvents = true;
}
// Warn once at startup if the internal timeout is shorter than the phases it is meant to outlast. It is
@@ -2521,76 +2620,6 @@ export class BasicCrawler<
}
}
- /**
- * Wrapper around the crawling context's `enqueueLinks` method:
- * - Injects `crawlDepth` to each request being added based on the crawling context request.
- * - Combines the `enqueueLinks` options with the crawler configuration - the user options take precedence,
- * but the crawler limits are always enforced (the `limit` is capped by the remaining `maxRequestsPerCrawl`
- * budget and skipped requests are always reported to the crawler too).
- * @internal
- */
- protected async enqueueLinksWithCrawlDepth(
- options: SetRequired<EnqueueLinksOptions, 'urls'>,
- request: Request<Dictionary>,
- requestManager: IRequestManager,
- ): Promise<BatchAddRequestsResult> {
- const transformRequestFunctionWrapper: RequestTransform = (requestOptions) => {
- requestOptions.crawlDepth = request.crawlDepth + 1;
-
- if (this.maxCrawlDepth !== undefined && requestOptions.crawlDepth! > this.maxCrawlDepth) {
- // Setting `skippedReason` before returning `false` ensures that `reportSkippedRequests`
- // reports `'depth'` as the reason (via `request.skippedReason ?? reason` fallback),
- // rather than the generic `'transform'` reason.
- requestOptions.skippedReason = 'depth';
- return false;
- }
-
- // After injecting the crawlDepth, we call the user-provided transform function, if there is one.
- // Its return value is passed through as is, so a falsy one still skips the request.
- return options.transformRequestFunction
- ? options.transformRequestFunction(requestOptions)
- : requestOptions;
- };
-
- const limit = await this.calculateEnqueuedRequestLimit(options.limit);
-
- // Create a request-scoped callback that logs enqueueLimit once per request handler call
- // Only log if an explicit limit was passed to enqueueLinks (not the internal maxRequestsPerCrawl-derived limit)
- let loggedEnqueueLimitForThisRequest = false;
- const onSkippedRequest: SkippedRequestCallback = async (skippedOptions) => {
- if (skippedOptions.reason === 'enqueueLimit') {
- if (!loggedEnqueueLimitForThisRequest && options.limit !== undefined) {
- this.log.info(
- limit === options.limit
- ? `Skipping URLs in the handler for ${request.url} due to the enqueueLinks limit of ${options.limit}.`
- : `Skipping URLs in the handler for ${request.url} due to the remaining maxRequestsPerCrawl budget of ${limit}, which is lower than the enqueueLinks limit of ${options.limit}.`,
- );
- loggedEnqueueLimitForThisRequest = true;
- }
- }
-
- await this.handleSkippedRequest(skippedOptions);
- await options.onSkippedRequest?.(skippedOptions);
- };
-
- // `enqueueLinks` applies `options.label`/`options.userData` to every newly enqueued request, so a single
- // validation against the label's schema covers them all (a no-op unless the router declares a schema).
- await this.validateRequestUserData({ label: options.label, userData: options.userData });
-
- return await enqueueLinks({
- ...options,
-
- // The options below are merged with the user options, so an explicitly `undefined` value
- // (e.g. `enqueueLinks({ urls, limit: config.limit })`) cannot discard the crawler defaults ⤵
- requestManager: options.requestManager ?? requestManager,
- robotsTxtFile: options.robotsTxtFile ?? (await this.getRobotsTxtFileForUrl(request.url)),
- respectRobotsTxtFile: options.respectRobotsTxtFile ?? this.#respectRobotsTxtFile,
- onSkippedRequest,
- limit,
- transformRequestFunction: transformRequestFunctionWrapper,
- });
- }
-
/**
* Generator function that yields requests injected with the given crawl depth.
* @internal
@@ -2816,19 +2845,14 @@ export class BasicCrawler<
* To stop the crawler gracefully (waiting for all running requests to finish), use {@apilink BasicCrawler.stop|`crawler.stop()`} instead.
*/
async teardown(): Promise<void> {
- // When this crawler initialized the event manager, its close() call emits
- // the final persistence event after the crawler-specific state has been
- // saved. External event managers still need an explicit event here.
- if (!this.#ownsEventManager) {
- serviceLocator.getEventManager().emit(EventType.PERSIST_STATE, { isMigrating: false });
- }
+ serviceLocator.getEventManager().emit(EventType.PERSIST_STATE, { isMigrating: false });
- await this.#sessionPoolDep.ifOwned(async (pool) => pool.teardown({ persistState: this.#ownsEventManager }));
-
- if (this.#ownsEventManager) {
+ if (this.#closeEvents) {
await serviceLocator.getEventManager().close();
}
+ await this.#sessionPoolDep.ifOwned((pool) => pool.teardown());
+
await this.#autoscaledPool?.abort();
await this.#concurrencySystemDep?.ifOwned((system) => system.stop());
}
@@ -2909,7 +2933,7 @@ export interface CreateContextOptions {
proxyInfo?: ProxyInfo;
}
-export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions {}
+export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions, EnqueueUrlsOptions {}
export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {}
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index 6e3799704..b58dd67a3 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -1,24 +1,24 @@
import type {
+ AddRequestsBatchedResult,
BasicCrawlerOptions,
BasicCrawlingContext,
ContextMiddleware,
CrawlingContext,
EnqueueLinksOptions,
ErrorHandler,
+ ExtractLinksOptions,
GetUserDataFromRequest,
- IRequestManager,
LoadedRequest,
Request,
RequestHandler,
RouterHandler,
- SkippedRequestCallback,
} from '@crawlee/basic';
import {
BasicCrawler,
browserPoolCookieToToughCookie,
ContextPipeline,
cookieStringToToughCookie,
- enqueueLinks,
+ EnqueueStrategy,
NavigationSkippedError,
OwnedOrInjected,
parseArgument,
@@ -43,18 +43,9 @@ import type {
LaunchContext,
} from '@crawlee/browser-pool';
import { BrowserPool, RemoteBrowserPool } from '@crawlee/browser-pool';
-import type {
- Awaitable,
- BatchAddRequestsResult,
- Cookie as CookieObject,
- Dictionary,
- IBrowserPool,
- ISession,
-} from '@crawlee/types';
-import type { RobotsTxtFile } from '@crawlee/utils';
+import type { Awaitable, Cookie as CookieObject, Dictionary, IBrowserPool, ISession } from '@crawlee/types';
import { CLOUDFLARE_RETRY_CSS_SELECTORS, RETRY_CSS_SELECTORS } from '@crawlee/utils/internal';
import { sleep } from '@crawlee/utils';
-import type { ReadonlyDeep } from 'type-fest';
import { z } from 'zod';
import { addTimeoutToPromise, TimeoutError, tryCancel } from '@apify/timeout';
@@ -103,10 +94,15 @@ export interface BrowserCrawlingContext<
*/
gotoOptions: GoToOptions;
+ /**
+ * Extracts URLs from the current page, without adding them to the request queue.
+ */
+ extractLinks: (options?: ExtractLinksOptions) => Promise<string[]>;
+
/**
* Helper function for extracting URLs from the current page and adding them to the request queue.
*/
- enqueueLinks: (options?: EnqueueLinksOptions) => Promise<BatchAddRequestsResult>;
+ enqueueLinks: (options?: EnqueueLinksOptions) => Promise<AddRequestsBatchedResult>;
}
export type BrowserHook<Context = BrowserCrawlingContext, ContextExtension = {}> = (
@@ -604,7 +600,15 @@ export abstract class BrowserCrawler<
});
tryCancel();
- const contextEnqueueLinks = crawlingContext.enqueueLinks;
+ const addRequests = crawlingContext.addRequests;
+
+ const extractLinks = async (options?: ExtractLinksOptions): Promise<string[]> => {
+ return extractUrlsFromPage(
+ page as any,
+ options?.selector ?? 'a',
+ options?.baseUrl ?? crawlingContext.request.loadedUrl ?? crawlingContext.request.url,
+ );
+ };
return {
page,
@@ -616,19 +620,22 @@ export abstract class BrowserCrawler<
get gotoOptions(): Dictionary {
throw new Error('The `gotoOptions` property is not available until `prepareNavigation` runs.');
},
- enqueueLinks: async (enqueueOptions: EnqueueLinksOptions = {}) => {
- return (await browserCrawlerEnqueueLinks({
- // `contextEnqueueLinks` clamps `limit` by the remaining `maxRequestsPerCrawl` budget itself;
- // pre-clamping it here would make the crawler log the internal limit as a user-provided one
- options: enqueueOptions,
- page,
- requestManager: await this.getRequestManager(),
- robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
- onSkippedRequest: this.handleSkippedRequest,
- originalRequestUrl: crawlingContext.request.url,
+ extractLinks,
+ enqueueLinks: async (options: EnqueueLinksOptions = {}) => {
+ const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
+ enqueueStrategy: options.strategy,
finalRequestUrl: crawlingContext.request.loadedUrl,
- enqueueLinks: contextEnqueueLinks,
- })) as BatchAddRequestsResult; // TODO make this type safe
+ originalRequestUrl: crawlingContext.request.url,
+ userProvidedBaseUrl: options.baseUrl,
+ });
+
+ const urls = await extractLinks(options);
+
+ return addRequests(urls, {
+ ...options,
+ baseUrl,
+ strategy: options.strategy ?? EnqueueStrategy.SameHostname,
+ });
},
};
}
@@ -881,70 +888,6 @@ export abstract class BrowserCrawler<
}
}
-/** @internal */
-interface EnqueueLinksInternalOptions {
- options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestManager'>> & Pick<EnqueueLinksOptions, 'requestManager'>;
- page: CommonPage;
- requestManager: IRequestManager;
- robotsTxtFile?: RobotsTxtFile;
- onSkippedRequest?: SkippedRequestCallback;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-/** @internal */
-interface BoundEnqueueLinksInternalOptions {
- enqueueLinks: BasicCrawlingContext['enqueueLinks'];
- options?: ReadonlyDeep<Omit<EnqueueLinksOptions, 'requestManager'>> & Pick<EnqueueLinksOptions, 'requestManager'>;
- originalRequestUrl: string;
- finalRequestUrl?: string;
- page: CommonPage;
-}
-
-/** @internal */
-function containsEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-): options is BoundEnqueueLinksInternalOptions {
- return !!(options as BoundEnqueueLinksInternalOptions).enqueueLinks;
-}
-
-/** @internal */
-export async function browserCrawlerEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-) {
- const { options: enqueueLinksOptions, finalRequestUrl, originalRequestUrl, page } = options;
-
- const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
- enqueueStrategy: enqueueLinksOptions?.strategy,
- finalRequestUrl,
- originalRequestUrl,
- userProvidedBaseUrl: enqueueLinksOptions?.baseUrl,
- });
-
- const urls = await extractUrlsFromPage(
- page as any,
- enqueueLinksOptions?.selector ?? 'a',
- enqueueLinksOptions?.baseUrl ?? finalRequestUrl ?? originalRequestUrl,
- );
-
- if (containsEnqueueLinks(options)) {
- return options.enqueueLinks({
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
- }
-
- return enqueueLinks({
- requestManager: options.requestManager,
- robotsTxtFile: options.robotsTxtFile,
- onSkippedRequest: options.onSkippedRequest,
- urls,
- ...(enqueueLinksOptions as EnqueueLinksOptions),
- baseUrl,
- });
-}
-
/**
* Extracts URLs from a given page.
* @ignore
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index 4ec3dc7d5..f2ab7eb1b 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -1,30 +1,28 @@
import type {
- BasicCrawlingContext,
+ AddRequestsBatchedResult,
CrawlingContext,
EnqueueLinksOptions,
ErrorHandler,
+ ExtractLinksOptions,
GetUserDataFromRequest,
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
- IRequestManager,
RequestHandler,
RouterHandler,
RouterRoutes,
RouteSchemas,
RoutesFromSchemas,
- SkippedRequestCallback,
} from '@crawlee/http';
import {
- enqueueLinks,
+ EnqueueStrategy,
HttpCrawler,
NavigationSkippedError,
resolveBaseUrlForEnqueueLinksFiltering,
Router,
} from '@crawlee/http';
-import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
+import type { Dictionary } from '@crawlee/types';
import { type CheerioRoot, extractUrlsFromCheerio } from '@crawlee/utils/internal';
-import { type RobotsTxtFile } from '@crawlee/utils';
import type { CheerioAPI, CheerioOptions } from 'cheerio';
import * as cheerio from 'cheerio';
import { parseDocument } from 'htmlparser2';
@@ -93,10 +91,15 @@ export interface CheerioCrawlingContext<
*/
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
+ /**
+ * Extracts URLs from the parsed HTML, without adding them to the request queue.
+ */
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
+
/**
* Helper function for extracting URLs from the parsed HTML and adding them to the request queue.
*/
- enqueueLinks(options?: EnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
}
export type CheerioRequestHandler<
@@ -250,22 +253,37 @@ export class CheerioCrawler<
}
private async addHelpers(crawlingContext: InternalHttpCrawlingContext & { $: CheerioAPI }) {
- const originalEnqueueLinks = crawlingContext.enqueueLinks;
+ const addRequests = crawlingContext.addRequests;
+
+ const extractLinks = async (options?: ExtractLinksOptions): Promise<string[]> => {
+ if (!crawlingContext.$) {
+ throw new Error('Cannot extract links because the DOM is not available.');
+ }
+
+ return extractUrlsFromCheerio(
+ crawlingContext.$,
+ options?.selector ?? 'a',
+ options?.baseUrl ?? crawlingContext.request.loadedUrl ?? crawlingContext.request.url,
+ );
+ };
return {
- enqueueLinks: async (enqueueOptions?: EnqueueLinksOptions) => {
- return (await cheerioCrawlerEnqueueLinks({
- // `originalEnqueueLinks` clamps `limit` by the remaining `maxRequestsPerCrawl` budget itself;
- // pre-clamping it here would make the crawler log the internal limit as a user-provided one
- options: enqueueOptions,
- $: crawlingContext.$,
- requestManager: await this.getRequestManager(),
- robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
- onSkippedRequest: this.handleSkippedRequest,
- originalRequestUrl: crawlingContext.request.url,
+ extractLinks,
+ enqueueLinks: async (options: EnqueueLinksOptions = {}) => {
+ const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
+ enqueueStrategy: options.strategy,
finalRequestUrl: crawlingContext.request.loadedUrl,
- enqueueLinks: originalEnqueueLinks,
- })) as BatchAddRequestsResult; // TODO make this type safe, see https://github.com/apify/crawlee/issues/4024
+ originalRequestUrl: crawlingContext.request.url,
+ userProvidedBaseUrl: options.baseUrl,
+ });
+
+ const urls = await extractLinks(options);
+
+ return addRequests(urls, {
+ ...options,
+ baseUrl,
+ strategy: options.strategy ?? EnqueueStrategy.SameHostname,
+ });
},
waitForSelector: async (selector: string, _timeoutMs?: number) => {
if (crawlingContext.$(selector).get().length === 0) {
@@ -283,70 +301,6 @@ export class CheerioCrawler<
}
}
-interface EnqueueLinksInternalOptions {
- options?: EnqueueLinksOptions;
- $: cheerio.CheerioAPI | null;
- requestManager: IRequestManager;
- robotsTxtFile?: RobotsTxtFile;
- onSkippedRequest?: SkippedRequestCallback;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-interface BoundEnqueueLinksInternalOptions {
- enqueueLinks: BasicCrawlingContext['enqueueLinks'];
- options?: EnqueueLinksOptions;
- $: cheerio.CheerioAPI | null;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-/** @internal */
-function containsEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-): options is BoundEnqueueLinksInternalOptions {
- return !!(options as BoundEnqueueLinksInternalOptions).enqueueLinks;
-}
-
-/** @internal */
-export async function cheerioCrawlerEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-) {
- const { options: enqueueLinksOptions, $, originalRequestUrl, finalRequestUrl } = options;
- if (!$) {
- throw new Error('Cannot enqueue links because the DOM is not available.');
- }
-
- const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
- enqueueStrategy: enqueueLinksOptions?.strategy,
- finalRequestUrl,
- originalRequestUrl,
- userProvidedBaseUrl: enqueueLinksOptions?.baseUrl,
- });
-
- const urls = extractUrlsFromCheerio(
- $,
- enqueueLinksOptions?.selector ?? 'a',
- enqueueLinksOptions?.baseUrl ?? finalRequestUrl ?? originalRequestUrl,
- );
-
- if (containsEnqueueLinks(options)) {
- return options.enqueueLinks({
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
- }
- return enqueueLinks({
- requestManager: options.requestManager,
- robotsTxtFile: options.robotsTxtFile,
- onSkippedRequest: options.onSkippedRequest,
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
-}
-
/**
* Creates new {@apilink Router} instance that works based on request labels.
* This instance can then serve as a `requestHandler` of your {@apilink CheerioCrawler}.
diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.ts
index 8edab7fa6..0420c81eb 100644
--- a/packages/core/src/crawlers/crawler_commons.ts
+++ b/packages/core/src/crawlers/crawler_commons.ts
@@ -1,13 +1,13 @@
import type { Dictionary, HttpRequestOptions, ISession, ProxyInfo, SendRequestOptions } from '@crawlee/types';
-import type { ReadonlyDeep, SetRequired } from 'type-fest';
+import type { ReadonlyDeep } from 'type-fest';
-import type { EnqueueLinksOptions } from '../enqueue_links/enqueue_links.js';
+import type { EnqueueUrlsOptions } from '../enqueue_links/enqueue_links.js';
import type { CrawleeLogger } from '../log.js';
import type { Request, RequestOptions, Source } from '../request.js';
import type { StorageIdentifier } from '../storages/storage_instance_manager.js';
import type { Dataset } from '../storages/dataset.js';
import type { KeyValueStore } from '../storages/key_value_store.js';
-import type { RequestQueueOperationOptions } from '../storages/request_queue.js';
+import type { AddRequestsBatchedResult } from '../storages/request_queue.js';
/** @internal */
export type IsAny<T> = 0 extends 1 & T ? true : false;
@@ -50,8 +50,8 @@ export type TypedRequestsLike<Routes extends Record<keyof Routes, Dictionary>> =
*/
export type TypedContextAddRequests<Routes extends Record<keyof Routes, Dictionary>> = (
requestsLike: ReadonlyDeep<LabeledSource<Routes>[]>,
- options?: ReadonlyDeep<RequestQueueOperationOptions>,
-) => Promise<void>;
+ options?: ReadonlyDeep<EnqueueUrlsOptions>,
+) => Promise<AddRequestsBatchedResult>;
/**
* An `enqueueLinks`-options object with its `label`/`userData` retyped according to a router's route map: a
@@ -120,42 +120,18 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction
): Promise<void>;
/**
- * This function automatically finds and enqueues links from the current page, adding them to the {@apilink RequestQueue}
- * currently used by the crawler.
+ * Add requests directly to the request queue currently used by the crawler.
*
- * Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.
- *
- * Check out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example
- * for more details regarding its usage.
- *
- * **Example usage**
- *
- * ```ts
- * async requestHandler({ enqueueLinks }) {
- * await enqueueLinks({
- * include: [
- * 'https://www.example.com/handbags/*',
- * ],
- * });
- * },
- * ```
- *
- * @param [options] All `enqueueLinks()` parameters are passed via an options object.
- */
- enqueueLinks: (
- options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>>,
- ) => Promise<unknown>;
-
- /**
- * Add requests directly to the request queue.
+ * Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns,
+ * the same way {@apilink CrawlingContext.enqueueLinks|`enqueueLinks`} does for extracted links.
*
* @param requests The requests to add
* @param options Options for the request queue
*/
addRequests: (
requestsLike: ReadonlyDeep<(string | Source)[]>,
- options?: ReadonlyDeep<RequestQueueOperationOptions>,
- ) => Promise<void>;
+ options?: ReadonlyDeep<EnqueueUrlsOptions>,
+ ) => Promise<AddRequestsBatchedResult>;
/**
* Returns the state - a piece of mutable persistent data shared across all the request handler runs.
@@ -176,35 +152,6 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction
}
export interface CrawlingContext<UserData extends Dictionary = Dictionary> extends RestrictedCrawlingContext<UserData> {
- /**
- * This function automatically finds and enqueues links from the current page, adding them to the {@apilink RequestQueue}
- * currently used by the crawler.
- *
- * Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.
- *
- * Check out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example
- * for more details regarding its usage.
- *
- * **Example usage**
- *
- * ```ts
- * async requestHandler({ enqueueLinks }) {
- * await enqueueLinks({
- * include: [
- * 'https://www.example.com/handbags/*',
- * ],
- * });
- * },
- * ```
- *
- * @param [options] All `enqueueLinks()` parameters are passed via an options object.
- * @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
- */
- enqueueLinks(
- options: ReadonlyDeep<Omit<SetRequired<EnqueueLinksOptions, 'urls'>, 'requestManager' | 'robotsTxtFile'>> &
- Pick<EnqueueLinksOptions, 'requestManager' | 'robotsTxtFile'>,
- ): Promise<unknown>;
-
/**
* Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.
*
diff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/core/src/enqueue_links/enqueue_links.ts
index d6a6ea634..e74205696 100644
--- a/packages/core/src/enqueue_links/enqueue_links.ts
+++ b/packages/core/src/enqueue_links/enqueue_links.ts
@@ -1,51 +1,30 @@
-import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
-import { EnqueueStrategy, type RobotsTxtFile } from '@crawlee/utils';
+import type { Dictionary } from '@crawlee/types';
+import { EnqueueStrategy } from '@crawlee/utils';
import { getDomain } from 'tldts';
-import type { SetRequired } from 'type-fest';
-import { z } from 'zod';
-import type { RequestOptions } from '../request.js';
-import { Request } from '../request.js';
-import type { IRequestManager } from '../storages/request_manager.js';
-import type {
- AddRequestsBatchedOptions,
- AddRequestsBatchedResult,
- RequestQueueOperationOptions,
-} from '../storages/request_queue.js';
-import { parseArgument, schemas } from '../validators.js';
-import type {
- RequestTransform,
- SkippedRequestCallback,
- SkippedRequestReason,
- UrlPatternInput,
- UrlPatternObject,
-} from './shared.js';
-import {
- applyRequestTransform,
- constructUrlPatternObjects,
- createRequestOptions,
- filterRequestOptionsByPatterns,
- urlPatternSchema,
-} from './shared.js';
+import type { RequestQueueOperationOptions } from '../storages/request_queue.js';
+import type { RequestTransform, SkippedRequestCallback, UrlPatternInput, UrlPatternObject } from './shared.js';
-export { EnqueueStrategy };
+/**
+ * Options shared by the `extractLinks()` context helper across crawler types.
+ */
+export interface ExtractLinksOptions {
+ /** A CSS selector matching links to be extracted. */
+ selector?: string;
-export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
/**
- * Limit the amount of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.
- * When called from a crawler context, the limit is further capped by what's left of the crawler's
- * {@apilink BasicCrawlerOptions.maxRequestsPerCrawl|`maxRequestsPerCrawl`} budget.
+ * A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,
+ * since the relative URL resolution is done inside the browser automatically.
*/
- limit?: number;
-
- /** An array of URLs to enqueue. */
- urls?: readonly string[];
-
- /** A request manager to which the URLs will be enqueued. */
- requestManager?: IRequestManager;
+ baseUrl?: string;
+}
- /** A CSS selector matching links to be enqueued. */
- selector?: string;
+/**
+ * Options accepted by the `enqueueUrls()` context helper exposed by `BasicCrawler`.
+ */
+export interface EnqueueUrlsOptions extends RequestQueueOperationOptions {
+ /** Limit the amount of actually enqueued URLs to this number. Useful for testing across the entire crawling scope. */
+ limit?: number;
/** Sets {@apilink Request.userData} for newly enqueued requests. */
userData?: Dictionary;
@@ -67,8 +46,7 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
skipNavigation?: boolean;
/**
- * A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer,
- * since the relative URL resolution is done inside the browser automatically.
+ * A base URL that will be used to resolve relative URLs.
*/
baseUrl?: string;
@@ -80,11 +58,11 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
* Glob matching is always case-insensitive.
* If you need case-sensitive matching, use a `RegExp`.
*
- * The patterns are combined with the {@apilink EnqueueLinksOptions.strategy|`strategy`} using AND logic - a URL
+ * The patterns are combined with the {@apilink EnqueueUrlsOptions.strategy|`strategy`} using AND logic - a URL
* must match at least one `include` pattern **and** satisfy the strategy to be enqueued. To match URLs across
* hostnames, pass an explicit {@apilink EnqueueStrategy.All} strategy.
*
- * If `undefined`, the links are enqueued based on the {@apilink EnqueueLinksOptions.strategy|`strategy`} alone.
+ * If `undefined`, the links are enqueued based on the {@apilink EnqueueUrlsOptions.strategy|`strategy`} alone.
* Passing an empty array is not allowed.
*/
include?: readonly UrlPatternInput[];
@@ -147,7 +125,7 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
*
* @default EnqueueStrategy.SameHostname
*/
- strategy?: EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
+ strategy?: EnqueueStrategyOption;
/**
* By default, only the first batch (1000) of found requests will be added to the queue before resolving the call.
@@ -155,267 +133,23 @@ export interface EnqueueLinksOptions extends RequestQueueOperationOptions {
*/
waitForAllRequestsToBeAdded?: boolean;
- /**
- * RobotsTxtFile instance for the current request that triggered the `enqueueLinks`.
- * If provided, disallowed URLs will be ignored.
- */
- robotsTxtFile?: Pick<RobotsTxtFile, 'isAllowed'>;
-
- /**
- * Mirrors {@apilink BasicCrawlerOptions.respectRobotsTxtFile}: pass `false` to disable filtering or
- * `{ userAgent }` to evaluate rules for a specific user-agent. Defaults to `*` when
- * {@apilink EnqueueLinksOptions.robotsTxtFile|`robotsTxtFile`} is provided.
- */
- respectRobotsTxtFile?: boolean | { userAgent?: string };
-
/**
* When a request is skipped for some reason, you can use this callback to act on it.
* This is currently fired for requests skipped
* 1. based on robots.txt file,
* 2. because they don't match enqueueLinks filters,
* 3. or because the maxRequestsPerCrawl limit has been reached
- *
- * When calling `enqueueLinks` through a crawler context, this callback runs in addition to (after) the
- * crawler-level `onSkippedRequest`, it does not replace it.
*/
onSkippedRequest?: SkippedRequestCallback;
}
-// `schemas.anyObject` passes values through by reference (object schemas return a pruned plain
-// copy), so `userData` keeps its identity for the enqueued requests.
-const enqueueLinksOptionsSchema = z.strictObject({
- urls: schemas.arrayOf(z.string(), 'strings'),
- requestManager: schemas.objectWithKeys(['addRequestsBatched']),
- robotsTxtFile: schemas.objectWithKeys(['isAllowed']).optional(),
- respectRobotsTxtFile: z.union([z.boolean(), z.strictObject({ userAgent: z.string().optional() })]).optional(),
- onSkippedRequest: schemas.anyFunction.optional(),
- forefront: z.boolean().optional(),
- skipNavigation: z.boolean().optional(),
- sessionId: z.string().optional(),
- limit: schemas.anyNumber.optional(),
- selector: z.string().optional(),
- baseUrl: z.string().optional(),
- userData: schemas.anyObject.optional(),
- label: z.string().optional(),
- include: schemas.arrayOf(urlPatternSchema, 'URL patterns').min(1).optional(),
- exclude: schemas.arrayOf(urlPatternSchema, 'URL patterns').optional(),
- transformRequestFunction: schemas.anyFunction.optional(),
- strategy: z.enum(EnqueueStrategy).optional(),
- waitForAllRequestsToBeAdded: z.boolean().optional(),
-});
-
-/**
- * This function enqueues the urls provided to the {@apilink RequestQueue} provided. If you want to automatically find and enqueue links,
- * you should use the context-aware `enqueueLinks` function provided on the crawler contexts.
- *
- * Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.
- *
- * **Example usage**
- *
- * ```javascript
- * await enqueueLinks({
- * urls: aListOfFoundUrls,
- * requestManager,
- * selector: 'a.product-detail',
- * include: [
- * 'https://www.example.com/handbags/*',
- * 'https://www.example.com/purses/*'
- * ],
- * });
- * ```
- *
- * @param options All `enqueueLinks()` parameters are passed via an options object.
- * @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
- */
-export async function enqueueLinks(
- options: SetRequired<Omit<EnqueueLinksOptions, 'requestManager'>, 'urls'> & {
- requestManager: {
- addRequestsBatched: (
- requests: Request<Dictionary>[],
- options: AddRequestsBatchedOptions,
- ) => Promise<AddRequestsBatchedResult>;
- };
- },
-): Promise<BatchAddRequestsResult> {
- if (!options || Object.keys(options).length === 0) {
- throw new RangeError(
- [
- 'enqueueLinks() was called without the required options. You can only do that when you use the `crawlingContext.enqueueLinks()` method in request handlers.',
- 'Check out our guide on how to use enqueueLinks() here: https://crawlee.dev/js/docs/examples/crawl-relative-links',
- ].join('\n'),
- );
- }
-
- const parsedOptions = parseArgument(options, enqueueLinksOptionsSchema, 'EnqueueLinksOptions');
-
- const {
- requestManager,
- limit,
- urls,
- include,
- exclude,
- transformRequestFunction,
- forefront,
- waitForAllRequestsToBeAdded,
- robotsTxtFile,
- onSkippedRequest,
- } = parsedOptions;
-
- const urlExcludePatternObjects: UrlPatternObject[] = exclude?.length ? constructUrlPatternObjects(exclude) : [];
- const urlPatternObjects: UrlPatternObject[] = include?.length ? constructUrlPatternObjects(include) : [];
-
- // The strategy always applies, even when `include` patterns are provided - the two are AND-ed together
- // (a URL must match an `include` pattern *and* satisfy the strategy). This mirrors crawlee-python.
- parsedOptions.strategy ??= EnqueueStrategy.SameHostname;
-
- const enqueueStrategyPatterns: UrlPatternObject[] = [];
-
- if (parsedOptions.baseUrl) {
- const url = new URL(parsedOptions.baseUrl);
-
- switch (parsedOptions.strategy) {
- case EnqueueStrategy.SameHostname:
- // We need to get the origin of the passed in domain in the event someone sets baseUrl
- // to an url like https://example.com/deep/default/path and one of the found urls is an
- // absolute relative path (/path/to/page)
- enqueueStrategyPatterns.push({ glob: ignoreHttpSchema(`${url.origin}/**`) });
- break;
- case EnqueueStrategy.SameDomain: {
- // Get the actual hostname from the base url
- const baseUrlHostname = getDomain(url.hostname, { mixedInputs: false });
-
- if (baseUrlHostname) {
- // We have a hostname, so we can use it to match all links on the page that point to it and any subdomains of it
- url.hostname = baseUrlHostname;
- enqueueStrategyPatterns.push(
- { glob: ignoreHttpSchema(`${url.origin.replace(baseUrlHostname, `*.${baseUrlHostname}`)}/**`) },
- { glob: ignoreHttpSchema(`${url.origin}/**`) },
- );
- } else {
- // We don't have a hostname (can happen for ips for instance), so reproduce the same behavior
- // as SameDomainAndSubdomain
- enqueueStrategyPatterns.push({ glob: ignoreHttpSchema(`${url.origin}/**`) });
- }
+/** The combined options accepted by a crawler context's `enqueueLinks()` helper: `extractLinks()` + `enqueueUrls()`. */
+export type EnqueueLinksOptions = ExtractLinksOptions & EnqueueUrlsOptions;
- break;
- }
- case EnqueueStrategy.SameOrigin: {
- // The same behavior as SameHostname, but respecting the protocol of the URL
- enqueueStrategyPatterns.push({ glob: `${url.origin}/**` });
- break;
- }
- case EnqueueStrategy.All:
- default:
- enqueueStrategyPatterns.push({ glob: `http{s,}://**` });
- break;
- }
- }
-
- async function reportSkippedRequests(
- skippedRequests: { url: string; skippedReason?: SkippedRequestReason }[],
- reason: SkippedRequestReason,
- ) {
- if (onSkippedRequest && skippedRequests.length > 0) {
- await Promise.all(
- skippedRequests.map((request) => {
- return onSkippedRequest({
- url: request.url,
- reason: request.skippedReason ?? reason,
- }) as Promise<void>;
- }),
- );
- }
- }
-
- let requestOptions = createRequestOptions(urls, parsedOptions);
-
- if (robotsTxtFile && parsedOptions.respectRobotsTxtFile !== false) {
- const robotsUserAgent =
- typeof parsedOptions.respectRobotsTxtFile === 'object'
- ? (parsedOptions.respectRobotsTxtFile.userAgent ?? '*')
- : '*';
- const skippedRequests: RequestOptions[] = [];
-
- requestOptions = requestOptions.filter((request) => {
- if (robotsTxtFile.isAllowed(request.url, robotsUserAgent)) {
- return true;
- }
-
- skippedRequests.push(request);
- return false;
- });
-
- await reportSkippedRequests(skippedRequests, 'robotsTxt');
- }
-
- async function createFilteredRequests() {
- const skippedRequests: string[] = [];
-
- // Step 1: Filter request options by exclude patterns, user include patterns, and strategy patterns.
- let filteredOptions: RequestOptions[];
- if (urlPatternObjects.length === 0) {
- filteredOptions = filterRequestOptionsByPatterns(
- requestOptions,
- enqueueStrategyPatterns.length > 0 ? enqueueStrategyPatterns : undefined,
- urlExcludePatternObjects,
- parsedOptions.strategy,
- (url) => skippedRequests.push(url),
- );
- } else {
- // Filter by user patterns first (with exclude)
- const afterUserPatterns = filterRequestOptionsByPatterns(
- requestOptions,
- urlPatternObjects,
- urlExcludePatternObjects,
- parsedOptions.strategy,
- (url) => skippedRequests.push(url),
- );
- // ...then filter by the enqueue links strategy (making this an AND check)
- filteredOptions = filterRequestOptionsByPatterns(
- afterUserPatterns,
- enqueueStrategyPatterns.length > 0 ? enqueueStrategyPatterns : undefined,
- [],
- parsedOptions.strategy,
- (url) => skippedRequests.push(url),
- );
- }
-
- await reportSkippedRequests(
- skippedRequests.map((url) => ({ url })),
- 'filters',
- );
-
- // Step 2: Apply transformRequestFunction on request options - it has the highest priority
- if (transformRequestFunction) {
- const skippedByTransform: RequestOptions[] = [];
- filteredOptions = applyRequestTransform(filteredOptions, transformRequestFunction, (r) =>
- skippedByTransform.push(r),
- );
- await reportSkippedRequests(skippedByTransform, 'transform');
- }
-
- // Step 3: Create Request instances from the final request options
- return filteredOptions.map((opts) => new Request(opts));
- }
-
- const { addedRequests, requestsOverLimit } = await requestManager.addRequestsBatched(
- await createFilteredRequests(),
- {
- forefront,
- waitForAllRequestsToBeAdded,
- maxNewRequests: limit,
- },
- );
-
- if (requestsOverLimit?.length !== undefined && requestsOverLimit.length > 0) {
- await reportSkippedRequests(
- requestsOverLimit.map((r) => ({ url: typeof r === 'string' ? r : r.url! })),
- 'enqueueLimit',
- );
- }
+export { EnqueueStrategy };
- return { processedRequests: addedRequests, unprocessedRequests: [] };
-}
+/** The `strategy` option accepted by {@apilink ExtractLinksOptions} and {@apilink EnqueueUrlsOptions}. */
+export type EnqueueStrategyOption = EnqueueStrategy | 'all' | 'same-domain' | 'same-hostname' | 'same-origin';
/**
* @internal
@@ -445,8 +179,8 @@ export function resolveBaseUrlForEnqueueLinksFiltering({
}
// If the user wants to ensure the same domain is accessed, regardless of subdomains, we check to ensure the domains match
- // If they don't (we went off domain via a redirect), we keep filtering against the original domain - returning
- // undefined here would disable the filtering entirely and enqueue every link on the redirected page
+ // Returning undefined here is intentional! If the domains don't match, having no baseUrl in enqueueLinks will cause it to not enqueue anything
+ // which is the intended behavior (since we went off domain)
if (enqueueStrategy === EnqueueStrategy.SameDomain) {
const originalHostname = getDomain(originalUrlOrigin, { mixedInputs: false })!;
const finalHostname = getDomain(finalUrlOrigin, { mixedInputs: false })!;
@@ -455,7 +189,7 @@ export function resolveBaseUrlForEnqueueLinksFiltering({
return finalUrlOrigin;
}
- return originalUrlOrigin;
+ return undefined;
}
// Always enqueue urls that are from the same origin in all other cases, as the filtering happens on the original request url, even if there was a redirect
@@ -468,11 +202,50 @@ export function resolveBaseUrlForEnqueueLinksFiltering({
*/
export interface ResolveBaseUrl {
userProvidedBaseUrl?: string;
- enqueueStrategy?: EnqueueLinksOptions['strategy'];
+ enqueueStrategy?: EnqueueStrategyOption;
originalRequestUrl: string;
finalRequestUrl?: string;
}
+/**
+ * @internal
+ * Builds the glob patterns a URL must match to satisfy the given enqueue `strategy`, anchored at `baseUrl`.
+ */
+export function buildEnqueueStrategyPatterns(baseUrl: string, strategy: EnqueueStrategyOption): UrlPatternObject[] {
+ const url = new URL(baseUrl);
+
+ switch (strategy) {
+ case EnqueueStrategy.SameHostname:
+ // We need to get the origin of the passed in domain in the event someone sets baseUrl
+ // to an url like https://example.com/deep/default/path and one of the found urls is an
+ // absolute relative path (/path/to/page)
+ return [{ glob: ignoreHttpSchema(`${url.origin}/**`) }];
+ case EnqueueStrategy.SameDomain: {
+ // Get the actual hostname from the base url
+ const baseUrlHostname = getDomain(url.hostname, { mixedInputs: false });
+
+ if (baseUrlHostname) {
+ // We have a hostname, so we can use it to match all links on the page that point to it and any subdomains of it
+ url.hostname = baseUrlHostname;
+ return [
+ { glob: ignoreHttpSchema(`${url.origin.replace(baseUrlHostname, `*.${baseUrlHostname}`)}/**`) },
+ { glob: ignoreHttpSchema(`${url.origin}/**`) },
+ ];
+ }
+
+ // We don't have a hostname (can happen for ips for instance), so reproduce the same behavior
+ // as SameDomainAndSubdomain
+ return [{ glob: ignoreHttpSchema(`${url.origin}/**`) }];
+ }
+ case EnqueueStrategy.SameOrigin:
+ // The same behavior as SameHostname, but respecting the protocol of the URL
+ return [{ glob: `${url.origin}/**` }];
+ case EnqueueStrategy.All:
+ default:
+ return [{ glob: `http{s,}://**` }];
+ }
+}
+
/**
* Internal function that changes the enqueue glob patterns to match both http and https
*/
diff --git a/packages/core/src/enqueue_links/shared.ts b/packages/core/src/enqueue_links/shared.ts
index b865ba43c..40798f770 100644
--- a/packages/core/src/enqueue_links/shared.ts
+++ b/packages/core/src/enqueue_links/shared.ts
@@ -1,12 +1,12 @@
import { URL } from 'node:url';
-import type { Awaitable } from '@crawlee/types';
+import type { Awaitable, Dictionary } from '@crawlee/types';
import { Minimatch } from 'minimatch';
import { z } from 'zod';
import type { RequestOptions } from '../request.js';
import { schemas } from '../validators.js';
-import type { EnqueueLinksOptions } from './enqueue_links.js';
+import type { EnqueueStrategyOption } from './enqueue_links.js';
export { tryAbsoluteURL } from '@crawlee/utils/internal';
@@ -177,7 +177,7 @@ export function filterRequestOptionsByPatterns(
requestOptions: RequestOptions[],
includePatterns: UrlPatternObject[] | undefined,
excludePatterns: UrlPatternObject[] = [],
- strategy?: EnqueueLinksOptions['strategy'],
+ strategy?: EnqueueStrategyOption,
onSkippedUrl?: (url: string) => void,
): RequestOptions[] {
const excludeMatchers = excludePatterns.map(createPatternObjectMatcher);
@@ -209,15 +209,29 @@ export function filterRequestOptionsByPatterns(
.filter((opts) => opts !== null);
}
+function isAbsoluteUrl(url: string): boolean {
+ try {
+ // eslint-disable-next-line no-new
+ new URL(url);
+ return true;
+ } catch {
+ return false;
+ }
+}
+
/**
* @ignore
*/
export function createRequestOptions(
sources: readonly (string | Record<string, unknown>)[],
- options: Pick<
- EnqueueLinksOptions,
- 'label' | 'userData' | 'baseUrl' | 'skipNavigation' | 'sessionId' | 'strategy'
- > = {},
+ options: {
+ label?: string;
+ userData?: Dictionary;
+ baseUrl?: string;
+ skipNavigation?: boolean;
+ sessionId?: string;
+ strategy?: EnqueueStrategyOption;
+ } = {},
): RequestOptions[] {
return sources
.map((src) =>
@@ -233,7 +247,12 @@ export function createRequestOptions(
}
})
.map((requestOptions) => {
- requestOptions.url = new URL(requestOptions.url, options.baseUrl).href;
+ // Leave already-absolute URLs untouched - re-deriving them via `new URL()` would normalize them
+ // (e.g. adding a trailing slash to a bare domain), which is surprising for URLs that didn't need
+ // resolving against `baseUrl` in the first place.
+ if (!isAbsoluteUrl(requestOptions.url)) {
+ requestOptions.url = new URL(requestOptions.url, options.baseUrl).href;
+ }
requestOptions.userData ??= options.userData ?? {};
if (typeof options.label === 'string') {
diff --git a/packages/core/src/request.ts b/packages/core/src/request.ts
index b80e84de6..d25c43ce9 100644
--- a/packages/core/src/request.ts
+++ b/packages/core/src/request.ts
@@ -7,7 +7,7 @@ import { z } from 'zod';
import { cryptoRandomObjectId, normalizeUrl } from '@apify/utilities';
-import type { EnqueueLinksOptions } from './enqueue_links/enqueue_links.js';
+import type { EnqueueStrategyOption } from './enqueue_links/enqueue_links.js';
import type { SkippedRequestReason } from './enqueue_links/shared.js';
import { serviceLocator } from './service_locator.js';
import { keys } from './typedefs.js';
@@ -417,11 +417,11 @@ class CrawleeRequest<UserData extends Dictionary = Dictionary> {
}
}
- private get enqueueStrategy(): EnqueueLinksOptions['strategy'] | undefined {
+ private get enqueueStrategy(): EnqueueStrategyOption | undefined {
return this.userData.__crawlee?.enqueueStrategy;
}
- private set enqueueStrategy(value: EnqueueLinksOptions['strategy']) {
+ private set enqueueStrategy(value: EnqueueStrategyOption | undefined) {
if (!this.userData.__crawlee) {
(this.userData as Dictionary).__crawlee = { enqueueStrategy: value };
} else {
@@ -651,7 +651,7 @@ export interface RequestOptions<UserData extends Dictionary = Dictionary> {
lockExpiresAt?: Date;
/** @internal */
- enqueueStrategy?: EnqueueLinksOptions['strategy'];
+ enqueueStrategy?: EnqueueStrategyOption;
}
export interface PushErrorMessageOptions {
diff --git a/packages/core/src/router.ts b/packages/core/src/router.ts
index 38a54bbdb..68af0c320 100644
--- a/packages/core/src/router.ts
+++ b/packages/core/src/router.ts
@@ -122,7 +122,7 @@ export type RouterLabel<Routes extends Record<keyof Routes, Dictionary>> = strin
: (keyof Routes & string) | symbol;
export interface RouterHandler<
- Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext,
+ Context extends RestrictedCrawlingContext = CrawlingContext,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
> extends Router<Context, Routes> {
(ctx: Context): Awaitable<void>;
@@ -276,7 +276,7 @@ export type RouterRoutes<Context, Routes extends Record<keyof Routes, Dictionary
* ```
*/
export class Router<
- Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'>,
+ Context extends RestrictedCrawlingContext,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
> {
readonly #routes: Map<string | symbol, (ctx: any) => Awaitable<void>> = new Map();
@@ -470,21 +470,21 @@ export class Router<
// treated as the legacy flat `userData` shape shared by all handlers. The third overload accepts a
// Standard Schema per label, inferring the route map and validating `userData` at runtime.
static create<
- Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext,
+ Context extends RestrictedCrawlingContext = CrawlingContext,
Routes extends Record<keyof Routes, Dictionary> = Record<string, GetUserDataFromRequest<Context['request']>>,
>(routes?: RouterRoutes<Context, Routes>): RouterHandler<Context, Routes>;
static create<
- Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext,
+ Context extends RestrictedCrawlingContext = CrawlingContext,
UserData extends Dictionary = GetUserDataFromRequest<Context['request']>,
>(routes?: RouterRoutes<Context, Record<string, UserData>>): RouterHandler<Context, Record<string, UserData>>;
static create<
- Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext,
+ Context extends RestrictedCrawlingContext = CrawlingContext,
const Schemas extends RouteSchemas = RouteSchemas,
>(schemas: Schemas): RouterHandler<Context, RoutesFromSchemas<Schemas>>;
- static create<Context extends Omit<RestrictedCrawlingContext, 'enqueueLinks'> = CrawlingContext>(
+ static create<Context extends RestrictedCrawlingContext = CrawlingContext>(
routesOrSchemas?: Record<string | symbol, ((ctx: any) => Awaitable<void>) | StandardSchemaV1>,
): RouterHandler<Context, any> {
const router = new Router<Context, any>();
diff --git a/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts b/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
index 1c74a4dcd..fa8049b81 100644
--- a/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
+++ b/packages/core/test/enqueue_links/add-enqueue-strategy-to-requests.test.ts
@@ -1,11 +1,13 @@
-import { load } from 'cheerio';
import {
- cheerioCrawlerEnqueueLinks,
+ CheerioCrawler,
EnqueueStrategy,
MemoryStorageBackend,
RequestQueue,
serviceLocator,
+ type Source,
} from 'crawlee';
+import { extractUrlsFromCheerio } from '@crawlee/utils/internal';
+import { load } from 'cheerio';
const HTML = `
<html>
@@ -25,23 +27,28 @@ const HTML = `
`;
describe('enqueueLinks() - it should store the enqueue strategy in requests', () => {
- const $ = load(HTML);
-
beforeEach(() => {
serviceLocator.setStorageBackend(new MemoryStorageBackend());
});
test('it should store the enqueue strategy in requests', async () => {
+ const enqueued: Source[] = [];
const requestQueue = await RequestQueue.open();
+ const originalAddRequests = requestQueue.addRequests.bind(requestQueue);
+ requestQueue.addRequests = async (requests, addOptions) => {
+ const items: Source[] = [];
+ for await (const request of requests) {
+ items.push(typeof request === 'string' ? { url: request } : (request as Source));
+ }
+ enqueued.push(...items);
+ return originalAddRequests(items, addOptions);
+ };
- await cheerioCrawlerEnqueueLinks({
- options: { selector: 'a', strategy: EnqueueStrategy.SameHostname },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://menicka.cz',
- });
+ const crawler = new CheerioCrawler({ requestManager: requestQueue });
+ const baseUrl = 'https://menicka.cz';
+ const urls = extractUrlsFromCheerio(load(HTML), 'a', baseUrl);
+ await crawler.addRequests(urls, { baseUrl, strategy: EnqueueStrategy.SameHostname });
- const enqueued = await requestQueue.fetchNextRequest();
- expect(enqueued!.userData.__crawlee.enqueueStrategy).toEqual(EnqueueStrategy.SameHostname);
+ expect((enqueued[0] as any).userData.__crawlee.enqueueStrategy).toEqual(EnqueueStrategy.SameHostname);
});
});
diff --git a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
index fac31d208..052bc1fb9 100644
--- a/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
+++ b/packages/core/test/enqueue_links/protocol-matching-based-on-strategy.test.ts
@@ -1,12 +1,13 @@
-import { load } from 'cheerio';
-import type { CheerioRoot } from 'crawlee';
import {
- cheerioCrawlerEnqueueLinks,
+ CheerioCrawler,
EnqueueStrategy,
MemoryStorageBackend,
RequestQueue,
serviceLocator,
+ type Source,
} from 'crawlee';
+import { extractUrlsFromCheerio } from '@crawlee/utils/internal';
+import { load } from 'cheerio';
const HTML = `
<html>
@@ -22,61 +23,50 @@ const HTML = `
</html>
`;
-type MemoryRequestQueueBackend = Awaited<ReturnType<MemoryStorageBackend['createRequestQueueBackend']>>;
+/**
+ * Runs `crawler.addRequests()` against the links extracted from `HTML`, resolved relative to `baseUrl` -
+ * the address `enqueueLinks()` would normally anchor the strategy to - and returns the resulting URLs.
+ */
+async function enqueuedUrls(strategy: EnqueueStrategy, baseUrl: string): Promise<Set<string>> {
+ const enqueued: Source[] = [];
+ const requestQueue = await RequestQueue.open();
+ const originalAddRequests = requestQueue.addRequests.bind(requestQueue);
+ requestQueue.addRequests = async (requests, addOptions) => {
+ const items: Source[] = [];
+ for await (const request of requests) {
+ items.push(typeof request === 'string' ? { url: request } : (request as Source));
+ }
+ enqueued.push(...items);
+ return originalAddRequests(items, addOptions);
+ };
+
+ const crawler = new CheerioCrawler({ requestManager: requestQueue });
+ const urls = extractUrlsFromCheerio(load(HTML), 'a', baseUrl);
+ await crawler.addRequests(urls, { baseUrl, strategy });
-/** Collect the URLs of all requests in the queue, regardless of order. */
-async function enqueuedUrls(requestQueue: RequestQueue): Promise<Set<string>> {
- const items = await (requestQueue.backend as MemoryRequestQueueBackend).listItems();
- return new Set(items.map((item) => item.url));
+ return new Set(enqueued.map((item) => item.url!));
}
describe('enqueueLinks() - matching and ignoring http/https protocol differences', () => {
- let $: CheerioRoot;
beforeEach(() => {
serviceLocator.setStorageBackend(new MemoryStorageBackend());
- $ = load(HTML);
});
test('SameHostname should ignore protocol difference', async () => {
- const requestQueue = await RequestQueue.open();
-
- await cheerioCrawlerEnqueueLinks({
- options: { selector: 'a', strategy: EnqueueStrategy.SameHostname },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(
+ expect(await enqueuedUrls(EnqueueStrategy.SameHostname, 'https://example.com')).toEqual(
new Set(['https://example.com/first', 'http://example.com/second']),
);
});
test('SameDomain should ignore protocol difference', async () => {
- const requestQueue = await RequestQueue.open();
-
- await cheerioCrawlerEnqueueLinks({
- options: { selector: 'a', strategy: EnqueueStrategy.SameDomain },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'http://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(
+ expect(await enqueuedUrls(EnqueueStrategy.SameDomain, 'http://example.com')).toEqual(
new Set(['https://example.com/first', 'http://example.com/second']),
);
});
test('SameOrigin should respect protocol', async () => {
- const requestQueue = await RequestQueue.open();
-
- await cheerioCrawlerEnqueueLinks({
- options: { selector: 'a', strategy: EnqueueStrategy.SameOrigin },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(new Set(['https://example.com/first']));
+ expect(await enqueuedUrls(EnqueueStrategy.SameOrigin, 'https://example.com')).toEqual(
+ new Set(['https://example.com/first']),
+ );
});
});
diff --git a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
index 705f7fd2f..23d1efeda 100644
--- a/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
+++ b/packages/core/test/enqueue_links/user-provided-patterns-with-enqueue-strategy.test.ts
@@ -1,12 +1,14 @@
-import { load } from 'cheerio';
-import type { CheerioRoot } from 'crawlee';
import {
- cheerioCrawlerEnqueueLinks,
+ CheerioCrawler,
+ type EnqueueUrlsOptions,
EnqueueStrategy,
MemoryStorageBackend,
RequestQueue,
serviceLocator,
+ type Source,
} from 'crawlee';
+import { extractUrlsFromCheerio } from '@crawlee/utils/internal';
+import { load } from 'cheerio';
const HTML = `
<html>
@@ -34,116 +36,71 @@ const HTML = `
</html>
`;
-type MemoryRequestQueueBackend = Awaited<ReturnType<MemoryStorageBackend['createRequestQueueBackend']>>;
-
-/** Collect the URLs of all requests in the queue, regardless of order. */
-async function enqueuedUrls(requestQueue: RequestQueue): Promise<Set<string>> {
- const items = await (requestQueue.backend as MemoryRequestQueueBackend).listItems();
- return new Set(items.map((item) => item.url));
+const SEED_URL = 'https://example.com';
+
+/**
+ * Runs `crawler.addRequests()` against the `.click` links extracted from `HTML`, resolved relative to
+ * `SEED_URL` - the address `enqueueLinks()` would normally anchor the strategy to - and returns the
+ * resulting URLs.
+ */
+async function enqueuedUrls(options: EnqueueUrlsOptions): Promise<Set<string>> {
+ const enqueued: Source[] = [];
+ const requestQueue = await RequestQueue.open();
+ const originalAddRequests = requestQueue.addRequests.bind(requestQueue);
+ requestQueue.addRequests = async (requests, addOptions) => {
+ const items: Source[] = [];
+ for await (const request of requests) {
+ items.push(typeof request === 'string' ? { url: request } : (request as Source));
+ }
+ enqueued.push(...items);
+ return originalAddRequests(items, addOptions);
+ };
+
+ const crawler = new CheerioCrawler({ requestManager: requestQueue });
+ const urls = extractUrlsFromCheerio(load(HTML), '.click', SEED_URL);
+ await crawler.addRequests(urls, { baseUrl: SEED_URL, ...options });
+
+ return new Set(enqueued.map((item) => item.url!));
}
describe('enqueueLinks() - combining user patterns with enqueue strategies', () => {
- let $: CheerioRoot;
beforeEach(() => {
serviceLocator.setStorageBackend(new MemoryStorageBackend());
- $ = load(HTML);
});
test('works with include and same domain strategy', async () => {
- const requestQueue = await RequestQueue.open();
-
const include = ['**/first'];
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- include,
- strategy: EnqueueStrategy.SameDomain,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(new Set(['https://example.com/a/b/first']));
+ expect(await enqueuedUrls({ include, strategy: EnqueueStrategy.SameDomain })).toEqual(
+ new Set(['https://example.com/a/b/first']),
+ );
});
test('works with include and all domains strategy', async () => {
- const requestQueue = await RequestQueue.open();
-
const include = ['**/first'];
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- include,
- strategy: EnqueueStrategy.All,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(
+ expect(await enqueuedUrls({ include, strategy: EnqueueStrategy.All })).toEqual(
new Set(['https://example.com/a/b/first', 'https://another.com/a/first']),
);
});
test('works with no user provided patterns but with same domain strategy', async () => {
- const requestQueue = await RequestQueue.open();
-
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.SameDomain,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(
+ expect(await enqueuedUrls({ strategy: EnqueueStrategy.SameDomain })).toEqual(
new Set(['https://example.com/a/b/first', 'https://example.com/a/b/third']),
);
});
test('works with include and exclude', async () => {
- const requestQueue = await RequestQueue.open();
-
const include = ['**/first'];
const exclude = ['**/first'];
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- include,
- exclude,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(new Set());
+ expect(await enqueuedUrls({ include, exclude })).toEqual(new Set());
});
test('works with exclude only', async () => {
- const requestQueue = await RequestQueue.open();
-
const exclude = ['**/second', '**/third', 'https://another.com/**'];
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- exclude,
- strategy: EnqueueStrategy.All,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(await enqueuedUrls(requestQueue)).toEqual(
+ expect(await enqueuedUrls({ exclude, strategy: EnqueueStrategy.All })).toEqual(
new Set(['https://example.com/a/b/first', 'http://cool.com/']),
);
});
diff --git a/packages/core/test/enqueue_links/userData.test.ts b/packages/core/test/enqueue_links/userData.test.ts
index ddd115dd4..4e3d15c22 100644
--- a/packages/core/test/enqueue_links/userData.test.ts
+++ b/packages/core/test/enqueue_links/userData.test.ts
@@ -1,5 +1,12 @@
-import { cheerioCrawlerEnqueueLinks, MemoryStorageBackend, RequestQueue, serviceLocator } from '@crawlee/cheerio';
-import type { CheerioAPI } from 'cheerio';
+import {
+ CheerioCrawler,
+ type EnqueueUrlsOptions,
+ MemoryStorageBackend,
+ RequestQueue,
+ serviceLocator,
+ type Source,
+} from '@crawlee/cheerio';
+import { extractUrlsFromCheerio } from '@crawlee/utils/internal';
import { load } from 'cheerio';
const HTML = `
@@ -16,48 +23,47 @@ const HTML = `
</html>
`;
-type MemoryRequestQueueBackend = Awaited<ReturnType<MemoryStorageBackend['createRequestQueueBackend']>>;
-
-/** Collect all requests currently in the queue (order not significant). */
-async function collect(requestQueue: RequestQueue) {
- return (requestQueue.backend as MemoryRequestQueueBackend).listItems();
+const BASE_URL = 'https://example.com';
+
+/**
+ * Runs `crawler.addRequests()` against the `selector`-matched links extracted from `HTML`, resolved relative
+ * to `BASE_URL`, and returns everything added.
+ */
+async function runEnqueueLinks(
+ { selector = 'a', ...options }: EnqueueUrlsOptions & { selector?: string },
+ requestManager: RequestQueue,
+): Promise<Source[]> {
+ const enqueued: Source[] = [];
+ const originalAddRequests = requestManager.addRequests.bind(requestManager);
+ requestManager.addRequests = async (requests, addOptions) => {
+ const items: Source[] = [];
+ for await (const request of requests) {
+ items.push(typeof request === 'string' ? { url: request } : (request as Source));
+ }
+ enqueued.push(...items);
+ return originalAddRequests(items, addOptions);
+ };
+
+ const crawler = new CheerioCrawler({ requestManager });
+ const urls = extractUrlsFromCheerio(load(HTML), selector, BASE_URL);
+ await crawler.addRequests(urls, { baseUrl: BASE_URL, ...options });
+
+ return enqueued;
}
describe("enqueueLinks() - userData shouldn't be changed and outer label must take priority", () => {
- let $: CheerioAPI;
beforeEach(() => {
serviceLocator.setStorageBackend(new MemoryStorageBackend());
- $ = load(HTML);
});
test('multiple enqueues with different labels', async () => {
const requestQueue = await RequestQueue.open();
const userData = { foo: 'bar' };
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: 'a.first',
- userData,
- label: 'first',
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: 'a.second',
- userData,
- label: 'second',
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ const first = await runEnqueueLinks({ selector: 'a.first', userData, label: 'first' }, requestQueue);
+ const second = await runEnqueueLinks({ selector: 'a.second', userData, label: 'second' }, requestQueue);
- const enqueued = await collect(requestQueue);
- const byUrl = Object.fromEntries(enqueued.map((r) => [r.url, r.userData?.label]));
+ const byUrl = Object.fromEntries([...first, ...second].map((r) => [r.url, (r as any).userData?.label]));
expect(byUrl).toEqual({
'https://example.com/first': 'first',
'https://example.com/second': 'second',
@@ -69,54 +75,30 @@ describe("enqueueLinks() - userData shouldn't be changed and outer label must ta
const userData = { foo: 'bar', label: 'bogus' };
const originalUserData = JSON.stringify(userData);
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: 'a.first',
- userData,
- label: 'first',
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ const enqueued = await runEnqueueLinks({ selector: 'a.first', userData, label: 'first' }, requestQueue);
const userDataAfterEnqueue = JSON.stringify(userData);
expect(userDataAfterEnqueue).toEqual(originalUserData);
- const enqueued = await collect(requestQueue);
expect(enqueued).toHaveLength(1);
expect(enqueued[0].url).toBe('https://example.com/first');
- expect(enqueued[0].userData?.label).toBe('first');
+ expect((enqueued[0] as any).userData?.label).toBe('first');
});
test('sets sessionId on all enqueued requests', async () => {
const requestQueue = await RequestQueue.open();
- await cheerioCrawlerEnqueueLinks({
- options: {
- sessionId: 'my-session',
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ const enqueued = await runEnqueueLinks({ sessionId: 'my-session' }, requestQueue);
- const enqueued = await collect(requestQueue);
expect(enqueued).toHaveLength(2);
- expect(enqueued.every((r) => r.userData?.__crawlee?.sessionId === 'my-session')).toBe(true);
+ expect(enqueued.every((r) => (r as any).userData?.__crawlee?.sessionId === 'my-session')).toBe(true);
});
test('does not set sessionId when option is not provided', async () => {
const requestQueue = await RequestQueue.open();
- await cheerioCrawlerEnqueueLinks({
- options: {},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ const enqueued = await runEnqueueLinks({}, requestQueue);
- const enqueued = await collect(requestQueue);
expect(enqueued).toHaveLength(2);
- expect(enqueued.every((r) => r.userData?.__crawlee?.sessionId === undefined)).toBe(true);
+ expect(enqueued.every((r) => (r as any).userData?.__crawlee?.sessionId === undefined)).toBe(true);
});
});
diff --git a/packages/crawlee/src/index.ts b/packages/crawlee/src/index.ts
index 0457cdef9..83940ac1d 100644
--- a/packages/crawlee/src/index.ts
+++ b/packages/crawlee/src/index.ts
@@ -1,4 +1,4 @@
-import { enqueueLinks, log } from '@crawlee/core';
+import { log } from '@crawlee/core';
import { playwrightUtils } from '@crawlee/playwright';
import { puppeteerUtils } from '@crawlee/puppeteer';
import { downloadListOfUrls, parseOpenGraph, sleep, social } from '@crawlee/utils';
@@ -20,7 +20,6 @@ export const utils = {
puppeteer: puppeteerUtils,
playwright: playwrightUtils,
log,
- enqueueLinks,
social,
sleep,
downloadListOfUrls,
diff --git a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
index ddde49b92..f67f1a595 100644
--- a/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
+++ b/packages/jsdom-crawler/src/internals/jsdom-crawler.ts
@@ -1,22 +1,21 @@
import type {
- BasicCrawlingContext,
+ AddRequestsBatchedResult,
CrawlingContext,
EnqueueLinksOptions,
ErrorHandler,
+ ExtractLinksOptions,
GetUserDataFromRequest,
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
- IRequestManager,
RequestHandler,
RouterHandler,
RouterRoutes,
RouteSchemas,
RoutesFromSchemas,
- SkippedRequestCallback,
} from '@crawlee/http';
import {
- enqueueLinks,
+ EnqueueStrategy,
HttpCrawler,
NavigationSkippedError,
parseArgument,
@@ -24,9 +23,9 @@ import {
Router,
tryAbsoluteURL,
} from '@crawlee/http';
-import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
+import type { Dictionary } from '@crawlee/types';
import { type CheerioRoot } from '@crawlee/utils/internal';
-import { type RobotsTxtFile, sleep } from '@crawlee/utils';
+import { sleep } from '@crawlee/utils';
import type { DOMWindow } from 'jsdom';
import { JSDOM, ResourceLoader, VirtualConsole } from 'jsdom';
import { z } from 'zod';
@@ -100,9 +99,14 @@ export interface JSDOMCrawlingContext<
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
/**
- * Helper function for extracting URLs from the parsed HTML and adding them to the request queue.
+ * Extracts URLs from the parsed DOM, without adding them to the request queue.
*/
- enqueueLinks(options?: EnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
+
+ /**
+ * Helper function for extracting URLs from the parsed DOM and adding them to the request queue.
+ */
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
}
export type JSDOMRequestHandler<
@@ -369,22 +373,37 @@ export class JSDOMCrawler<
}
private async addHelpers(crawlingContext: InternalHttpCrawlingContext & { body: string; window: DOMWindow }) {
- const originalEnqueueLinks = crawlingContext.enqueueLinks;
+ const addRequests = crawlingContext.addRequests;
+
+ const extractLinks = async (options?: ExtractLinksOptions): Promise<string[]> => {
+ if (!crawlingContext.window) {
+ throw new Error('Cannot extract links because the JSDOM is not available.');
+ }
+
+ return extractUrlsFromWindow(
+ crawlingContext.window,
+ options?.selector ?? 'a',
+ options?.baseUrl ?? crawlingContext.request.loadedUrl ?? crawlingContext.request.url,
+ );
+ };
return {
- enqueueLinks: async (enqueueOptions?: EnqueueLinksOptions) => {
- return (await domCrawlerEnqueueLinks({
- // `originalEnqueueLinks` clamps `limit` by the remaining `maxRequestsPerCrawl` budget itself;
- // pre-clamping it here would make the crawler log the internal limit as a user-provided one
- options: enqueueOptions,
- window: crawlingContext.window,
- requestManager: await this.getRequestManager(),
- robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
- onSkippedRequest: this.handleSkippedRequest,
- originalRequestUrl: crawlingContext.request.url,
+ extractLinks,
+ enqueueLinks: async (options: EnqueueLinksOptions = {}) => {
+ const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
+ enqueueStrategy: options.strategy,
finalRequestUrl: crawlingContext.request.loadedUrl,
- enqueueLinks: originalEnqueueLinks,
- })) as BatchAddRequestsResult; // TODO make this type safe, see https://github.com/apify/crawlee/issues/4024
+ originalRequestUrl: crawlingContext.request.url,
+ userProvidedBaseUrl: options.baseUrl,
+ });
+
+ const urls = await extractLinks(options);
+
+ return addRequests(urls, {
+ ...options,
+ baseUrl,
+ strategy: options.strategy ?? EnqueueStrategy.SameHostname,
+ });
},
async waitForSelector(selector: string, timeoutMs = 5_000) {
const cheerio = await import('cheerio');
@@ -414,70 +433,6 @@ export class JSDOMCrawler<
}
}
-interface EnqueueLinksInternalOptions {
- options?: EnqueueLinksOptions;
- window: DOMWindow | null;
- requestManager: IRequestManager;
- robotsTxtFile?: RobotsTxtFile;
- onSkippedRequest?: SkippedRequestCallback;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-interface BoundEnqueueLinksInternalOptions {
- enqueueLinks: BasicCrawlingContext['enqueueLinks'];
- options?: EnqueueLinksOptions;
- window: DOMWindow | null;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-/** @internal */
-function containsEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-): options is BoundEnqueueLinksInternalOptions {
- return !!(options as BoundEnqueueLinksInternalOptions).enqueueLinks;
-}
-
-/** @internal */
-export async function domCrawlerEnqueueLinks(options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions) {
- const { options: enqueueLinksOptions, window, originalRequestUrl, finalRequestUrl } = options;
-
- if (!window) {
- throw new Error('Cannot enqueue links because the JSDOM is not available.');
- }
-
- const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
- enqueueStrategy: enqueueLinksOptions?.strategy,
- finalRequestUrl,
- originalRequestUrl,
- userProvidedBaseUrl: enqueueLinksOptions?.baseUrl,
- });
-
- const urls = extractUrlsFromWindow(
- window,
- enqueueLinksOptions?.selector ?? 'a',
- enqueueLinksOptions?.baseUrl ?? finalRequestUrl ?? originalRequestUrl,
- );
-
- if (containsEnqueueLinks(options)) {
- return options.enqueueLinks({
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
- }
-
- return enqueueLinks({
- requestManager: options.requestManager,
- robotsTxtFile: options.robotsTxtFile,
- onSkippedRequest: options.onSkippedRequest,
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
-}
-
/**
* Extracts URLs from a given Window object.
* @ignore
diff --git a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
index cda4dc2d2..d9be04493 100644
--- a/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
+++ b/packages/linkedom-crawler/src/internals/linkedom-crawler.ts
@@ -1,31 +1,30 @@
import type {
- BasicCrawlingContext,
+ AddRequestsBatchedResult,
CrawlingContext,
EnqueueLinksOptions,
ErrorHandler,
+ ExtractLinksOptions,
GetUserDataFromRequest,
HttpCrawlerOptions,
InternalHttpCrawlingContext,
InternalHttpHook,
- IRequestManager,
RequestHandler,
RouterHandler,
RouterRoutes,
RouteSchemas,
RoutesFromSchemas,
- SkippedRequestCallback,
} from '@crawlee/http';
import {
- enqueueLinks,
+ EnqueueStrategy,
HttpCrawler,
NavigationSkippedError,
resolveBaseUrlForEnqueueLinksFiltering,
Router,
tryAbsoluteURL,
} from '@crawlee/http';
-import type { BatchAddRequestsResult, Dictionary } from '@crawlee/types';
+import type { Dictionary } from '@crawlee/types';
import { type CheerioRoot } from '@crawlee/utils/internal';
-import { type RobotsTxtFile, sleep } from '@crawlee/utils';
+import { sleep } from '@crawlee/utils';
import * as cheerio from 'cheerio';
import { DOMParser } from 'linkedom/cached';
@@ -43,8 +42,6 @@ export interface LinkeDOMCrawlerOptions<
Routes extends Record<keyof Routes, Dictionary> = Record<string, UserData>,
> extends HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes> {}
-export interface LinkeDOMCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOptions, 'urls' | 'requestManager'> {}
-
export type LinkeDOMHook<
UserData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
JSONData extends Dictionary = any, // with default to Dictionary we cant use a typed router in untyped crawler
@@ -92,9 +89,14 @@ export interface LinkeDOMCrawlingContext<
parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioRoot>;
/**
- * Helper function for extracting URLs from the parsed HTML and adding them to the request queue.
+ * Extracts URLs from the parsed DOM, without adding them to the request queue.
+ */
+ extractLinks(options?: ExtractLinksOptions): Promise<string[]>;
+
+ /**
+ * Helper function for extracting URLs from the parsed DOM and adding them to the request queue.
*/
- enqueueLinks(options?: LinkeDOMCrawlerEnqueueLinksOptions): Promise<BatchAddRequestsResult>;
+ enqueueLinks(options?: EnqueueLinksOptions): Promise<AddRequestsBatchedResult>;
}
export type LinkeDOMRequestHandler<
@@ -250,22 +252,37 @@ export class LinkeDOMCrawler<
}
private async addHelpers(crawlingContext: InternalHttpCrawlingContext & { body: string; window: Window }) {
- const originalEnqueueLinks = crawlingContext.enqueueLinks;
+ const addRequests = crawlingContext.addRequests;
+
+ const extractLinks = async (options?: ExtractLinksOptions): Promise<string[]> => {
+ if (!crawlingContext.window) {
+ throw new Error('Cannot extract links because the DOM is not available.');
+ }
+
+ return extractUrlsFromWindow(
+ crawlingContext.window,
+ options?.selector ?? 'a',
+ options?.baseUrl ?? crawlingContext.request.loadedUrl ?? crawlingContext.request.url,
+ );
+ };
return {
- enqueueLinks: async (enqueueOptions?: LinkeDOMCrawlerEnqueueLinksOptions) => {
- return (await linkedomCrawlerEnqueueLinks({
- // `originalEnqueueLinks` clamps `limit` by the remaining `maxRequestsPerCrawl` budget itself;
- // pre-clamping it here would make the crawler log the internal limit as a user-provided one
- options: enqueueOptions,
- window: crawlingContext.window,
- requestManager: await this.getRequestManager(),
- robotsTxtFile: await this.getRobotsTxtFileForUrl(crawlingContext.request.url),
- onSkippedRequest: this.handleSkippedRequest,
- originalRequestUrl: crawlingContext.request.url,
+ extractLinks,
+ enqueueLinks: async (options: EnqueueLinksOptions = {}) => {
+ const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
+ enqueueStrategy: options.strategy,
finalRequestUrl: crawlingContext.request.loadedUrl,
- enqueueLinks: originalEnqueueLinks,
- })) as BatchAddRequestsResult; // TODO make this type safe, see https://github.com/apify/crawlee/issues/4024
+ originalRequestUrl: crawlingContext.request.url,
+ userProvidedBaseUrl: options.baseUrl,
+ });
+
+ const urls = await extractLinks(options);
+
+ return addRequests(urls, {
+ ...options,
+ baseUrl,
+ strategy: options.strategy ?? EnqueueStrategy.SameHostname,
+ });
},
async waitForSelector(selector: string, timeoutMs = 5_000) {
const $ = cheerio.load(crawlingContext.body);
@@ -293,72 +310,6 @@ export class LinkeDOMCrawler<
}
}
-interface EnqueueLinksInternalOptions {
- options?: EnqueueLinksOptions;
- window: Window | null;
- requestManager: IRequestManager;
- robotsTxtFile?: RobotsTxtFile;
- onSkippedRequest?: SkippedRequestCallback;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-interface BoundEnqueueLinksInternalOptions {
- enqueueLinks: BasicCrawlingContext['enqueueLinks'];
- options?: EnqueueLinksOptions;
- window: Window | null;
- originalRequestUrl: string;
- finalRequestUrl?: string;
-}
-
-/** @internal */
-function containsEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-): options is BoundEnqueueLinksInternalOptions {
- return !!(options as BoundEnqueueLinksInternalOptions).enqueueLinks;
-}
-
-/** @internal */
-export async function linkedomCrawlerEnqueueLinks(
- options: EnqueueLinksInternalOptions | BoundEnqueueLinksInternalOptions,
-) {
- const { options: enqueueLinksOptions, window, originalRequestUrl, finalRequestUrl } = options;
-
- if (!window) {
- throw new Error('Cannot enqueue links because the DOM is not available.');
- }
-
- const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
- enqueueStrategy: enqueueLinksOptions?.strategy,
- finalRequestUrl,
- originalRequestUrl,
- userProvidedBaseUrl: enqueueLinksOptions?.baseUrl,
- });
-
- const urls = extractUrlsFromWindow(
- window,
- enqueueLinksOptions?.selector ?? 'a',
- enqueueLinksOptions?.baseUrl ?? finalRequestUrl ?? originalRequestUrl,
- );
-
- if (containsEnqueueLinks(options)) {
- return options.enqueueLinks({
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
- }
-
- return enqueueLinks({
- requestManager: options.requestManager,
- robotsTxtFile: options.robotsTxtFile,
- onSkippedRequest: options.onSkippedRequest,
- urls,
- ...enqueueLinksOptions,
- baseUrl,
- });
-}
-
/**
* Extracts URLs from a given Window object.
* @ignore
diff --git a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
index 866d73391..14f013d6f 100644
--- a/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts
@@ -14,6 +14,7 @@ import { extractUrlsFromPage } from '@crawlee/browser';
import type { CheerioCrawlingContext } from '@crawlee/cheerio';
import { CheerioCrawler } from '@crawlee/cheerio';
import type {
+ AddRequestsBatchedResult,
ContextPipeline,
CrawleeLogger,
CrawlingContext,
@@ -29,6 +30,7 @@ import type {
} from '@crawlee/core';
import {
createStorageTransaction,
+ EnqueueStrategy,
OwnedOrInjected,
parseArgument,
RequestHandlerError,
@@ -36,12 +38,11 @@ import {
Router,
Statistics,
} from '@crawlee/core';
-import type { BatchAddRequestsResult, Dictionary, Awaitable } from '@crawlee/types';
+import type { Dictionary, Awaitable } from '@crawlee/types';
import { type CheerioRoot, extractUrlsFromCheerio } from '@crawlee/utils/internal';
import { type Cheerio } from 'cheerio';
import type { AnyNode } from 'domhandler';
import type { Page } from 'playwright';
-import type { SetRequired } from 'type-fest';
import { z } from 'zod';
import { addTimeoutToPromise } from '@apify/timeout';
@@ -498,6 +499,9 @@ export class AdaptivePlaywrightCrawler<
get parseWithCheerio(): AdaptivePlaywrightCrawlerContext['parseWithCheerio'] {
throw new Error(errorMessage('parseWithCheerio'));
},
+ get enqueueLinks(): AdaptivePlaywrightCrawlerContext['enqueueLinks'] {
+ throw new Error(errorMessage('enqueueLinks'));
+ },
}),
});
}
@@ -514,14 +518,12 @@ export class AdaptivePlaywrightCrawler<
return cheerioContext.$(selector);
},
enqueueLinks: async (options: EnqueueLinksOptions = {}) => {
- const urls =
- options.urls ??
- extractUrlsFromCheerio(
- cheerioContext.$,
- options.selector,
- options.baseUrl ?? cheerioContext.request.loadedUrl,
- );
- return (await this.enqueueLinks({ ...options, urls }, cheerioContext.request)) as unknown as void;
+ const urls = extractUrlsFromCheerio(
+ cheerioContext.$,
+ options.selector,
+ options.baseUrl ?? cheerioContext.request.loadedUrl,
+ );
+ return (await this.enqueueLinks(urls, options, cheerioContext.request)) as unknown as void;
},
response: cheerioContext.response,
};
@@ -553,24 +555,16 @@ export class AdaptivePlaywrightCrawler<
},
enqueueLinks: async (options: EnqueueLinksOptions = {}, timeoutMs = 5000) => {
// TODO consider using `context.parseWithCheerio` to make this universal and avoid code duplication
- let urls: readonly string[];
-
- if (options.urls === undefined) {
- const selector = options.selector ?? 'a';
- const locator = playwrightContext.page.locator(selector).first();
- await locator.waitFor({ timeout: timeoutMs, state: 'attached' });
- urls =
- options.urls ??
- (await extractUrlsFromPage(
- playwrightContext.page,
- selector,
- options.baseUrl ?? playwrightContext.request.loadedUrl,
- ));
- } else {
- urls = options.urls;
- }
+ const selector = options.selector ?? 'a';
+ const locator = playwrightContext.page.locator(selector).first();
+ await locator.waitFor({ timeout: timeoutMs, state: 'attached' });
+ const urls = await extractUrlsFromPage(
+ playwrightContext.page,
+ selector,
+ options.baseUrl ?? playwrightContext.request.loadedUrl,
+ );
- return (await this.enqueueLinks({ ...options, urls }, playwrightContext.request)) as unknown as void;
+ return (await this.enqueueLinks(urls, options, playwrightContext.request)) as unknown as void;
},
};
}
@@ -788,9 +782,10 @@ export class AdaptivePlaywrightCrawler<
}
private async enqueueLinks(
- options: SetRequired<EnqueueLinksOptions, 'urls'>,
+ urls: readonly string[],
+ options: EnqueueLinksOptions,
request: RestrictedCrawlingContext['request'],
- ): Promise<BatchAddRequestsResult> {
+ ): Promise<AddRequestsBatchedResult> {
const baseUrl = resolveBaseUrlForEnqueueLinksFiltering({
enqueueStrategy: options?.strategy,
finalRequestUrl: request.loadedUrl,
@@ -798,9 +793,15 @@ export class AdaptivePlaywrightCrawler<
userProvidedBaseUrl: options?.baseUrl,
});
+ const requestsWithDepth = this.addCrawlDepthRequestGenerator(urls, request.crawlDepth + 1);
+
// The per-attempt transaction buffers these (the queue policy defaults to `deferred` here),
// so a discarded attempt's enqueues never reach the queue.
- return await this.enqueueLinksWithCrawlDepth({ ...options, baseUrl }, request, await this.getRequestManager());
+ return await this.addRequests(requestsWithDepth, {
+ ...options,
+ baseUrl,
+ strategy: options.strategy ?? EnqueueStrategy.SameHostname,
+ });
}
private createLogProxy(log: CrawleeLogger, logs: LogProxyCall[]) {
diff --git a/test/core/crawlers/adaptive_playwright_crawler.test.ts b/test/core/crawlers/adaptive_playwright_crawler.test.ts
index b16ea81d5..562a97271 100644
--- a/test/core/crawlers/adaptive_playwright_crawler.test.ts
+++ b/test/core/crawlers/adaptive_playwright_crawler.test.ts
@@ -606,8 +606,8 @@ describe('AdaptivePlaywrightCrawler', () => {
strategy: 'same-hostname',
});
- for (const processedRequest of result.processedRequests) {
- enqueuedUrls.add(processedRequest.uniqueKey);
+ for (const addedRequest of result.addedRequests) {
+ enqueuedUrls.add(addedRequest.uniqueKey);
}
}
},
diff --git a/test/core/crawlers/basic_crawler.test.ts b/test/core/crawlers/basic_crawler.test.ts
index 900052b55..bec9e6df2 100644
--- a/test/core/crawlers/basic_crawler.test.ts
+++ b/test/core/crawlers/basic_crawler.test.ts
@@ -40,7 +40,6 @@ import { BaseHttpClient } from '@crawlee/http-client';
import type { Dictionary, ISession, ProxyInfo } from '@crawlee/types';
import { RobotsTxtFile, sleep } from '@crawlee/utils';
import express from 'express';
-import type { SetRequired } from 'type-fest';
import type { Mock } from 'vitest';
import { afterAll, beforeAll, beforeEach, describe, expect, test, vitest } from 'vitest';
import { z } from 'zod';
@@ -459,14 +458,12 @@ describe('BasicCrawler', () => {
test('enqueueLinks should respect maxCrawlDepth', async () => {
const processedUrls: string[] = [];
- const requestHandler: RequestHandler = async ({ request, enqueueLinks }) => {
+ const requestHandler: RequestHandler = async ({ request, addRequests }) => {
processedUrls.push(request.url);
const url = new URL(request.url);
url.pathname = `${url.pathname}deep/`;
- await enqueueLinks({
- urls: [url.toString()],
- });
+ await addRequests([url.toString()]);
};
const crawler = new BasicCrawler({
@@ -484,57 +481,57 @@ describe('BasicCrawler', () => {
]);
});
- describe('enqueueLinksWithCrawlDepth()', () => {
+ describe('addRequests() filters', () => {
let onSkippedRequestMock: Mock;
let addRequestsBatchedMock: Mock;
- let options: SetRequired<EnqueueLinksOptions, 'urls'>;
- let request: Request;
+ let drainedRequests: any[];
+ let options: EnqueueLinksOptions;
let requestQueue: RequestQueue;
- type EnqueueLinksWrapperOptions = Parameters<BasicCrawler['enqueueLinksWithCrawlDepth']>;
- class TestCrawler extends BasicCrawler {
- public exposedEnqueueLinksWithCrawlDepth(...enqueueLinksOptions: EnqueueLinksWrapperOptions) {
- return this.enqueueLinksWithCrawlDepth(...enqueueLinksOptions);
- }
- }
+ const crawler = new BasicCrawler({ maxCrawlDepth: 3 });
- const crawler = new TestCrawler({ maxCrawlDepth: 3 });
+ // Mimics what `context.addRequests()` would have tagged the URLs with, based on the current
+ // request's `crawlDepth`.
+ const atDepth = (urls: string[], crawlDepth: number) => urls.map((url) => ({ url, crawlDepth }));
beforeEach(() => {
- addRequestsBatchedMock = vi.fn().mockImplementation(async () => ({
- addedRequests: [],
- waitForAllRequestsToBeAdded: Promise.resolve([]),
- requestsOverLimit: [],
- }));
+ drainedRequests = [];
+ // `addRequests()` now streams via an async generator (matching how it always fed
+ // `addRequestsBatched()`), so the mock has to drain it to see what was actually produced.
+ addRequestsBatchedMock = vi.fn().mockImplementation(async (requests: AsyncIterable<unknown>) => {
+ for await (const request of requests) {
+ drainedRequests.push(request);
+ }
+ return { addedRequests: [], waitForAllRequestsToBeAdded: Promise.resolve([]), requestsOverLimit: [] };
+ });
onSkippedRequestMock = vi.fn();
options = {
- urls: ['https://example.com/1/', 'https://example.com/2/'],
onSkippedRequest: onSkippedRequestMock,
};
- request = new Request({ url: 'https://example.com/', crawlDepth: 2 });
requestQueue = {
addRequestsBatched: addRequestsBatchedMock as RequestQueue['addRequestsBatched'],
} as RequestQueue;
+ // eslint-disable-next-line dot-notation -- private field on the crawler, injected for the mock
+ crawler['requestManager'] = requestQueue;
});
it('should generate requests with maxCrawlDepth', async () => {
- await crawler.exposedEnqueueLinksWithCrawlDepth(options, request, requestQueue);
+ const urls = atDepth(['https://example.com/1/', 'https://example.com/2/'], 3);
+ await crawler.addRequests(urls, options);
- const requests = addRequestsBatchedMock.mock.calls[0][0];
- expect(requests).toHaveLength(2);
- expect(requests[0]).toMatchObject({ url: 'https://example.com/1/', crawlDepth: 3 });
- expect(requests[1]).toMatchObject({ url: 'https://example.com/2/', crawlDepth: 3 });
+ expect(drainedRequests).toHaveLength(2);
+ expect(drainedRequests[0]).toMatchObject({ url: 'https://example.com/1/', crawlDepth: 3 });
+ expect(drainedRequests[1]).toMatchObject({ url: 'https://example.com/2/', crawlDepth: 3 });
expect(onSkippedRequestMock).not.toBeCalled();
});
it('should skip requests with crawlDepth exceeding maxCrawlDepth', async () => {
- const requestWithMaxDepth = new Request({ url: 'https://example.com/', crawlDepth: 3 });
- await crawler.exposedEnqueueLinksWithCrawlDepth(options, requestWithMaxDepth, requestQueue);
+ const urls = atDepth(['https://example.com/1/', 'https://example.com/2/'], 4);
+ await crawler.addRequests(urls, options);
- const requests = addRequestsBatchedMock.mock.calls[0][0];
- expect(requests).toHaveLength(0);
+ expect(drainedRequests).toHaveLength(0);
const skippedRequests = onSkippedRequestMock.mock.calls.map((call) => call[0]);
expect(skippedRequests).toHaveLength(2);
@@ -543,16 +540,16 @@ describe('BasicCrawler', () => {
});
it('should respect user provided transformRequestFunction', async () => {
+ const urls = atDepth(['https://example.com/1/', 'https://example.com/2/'], 3);
const transformRequestFunction = vi.fn((req: RequestOptions) => req);
const optionsWithTransform = { ...options, transformRequestFunction };
- await crawler.exposedEnqueueLinksWithCrawlDepth(optionsWithTransform, request, requestQueue);
+ await crawler.addRequests(urls, optionsWithTransform);
expect(transformRequestFunction).toHaveBeenCalled();
- const requests = addRequestsBatchedMock.mock.calls[0][0];
- expect(requests).toHaveLength(2);
- expect(requests[0]).toMatchObject({ url: 'https://example.com/1/', crawlDepth: 3 });
+ expect(drainedRequests).toHaveLength(2);
+ expect(drainedRequests[0]).toMatchObject({ url: 'https://example.com/1/', crawlDepth: 3 });
});
it.each([null, undefined, false] as const)(
@@ -574,14 +571,13 @@ describe('BasicCrawler', () => {
);
it('should report depth-limited requests with reason "depth" even when user transformRequestFunction is provided', async () => {
+ const urls = atDepth(['https://example.com/1/', 'https://example.com/2/'], 4);
const transformRequestFunction = vi.fn((req: RequestOptions) => req);
- const requestWithMaxDepth = new Request({ url: 'https://example.com/', crawlDepth: 3 });
const optionsWithTransform = { ...options, transformRequestFunction };
- await crawler.exposedEnqueueLinksWithCrawlDepth(optionsWithTransform, requestWithMaxDepth, requestQueue);
+ await crawler.addRequests(urls, optionsWithTransform);
- const requests = addRequestsBatchedMock.mock.calls[0][0];
- expect(requests).toHaveLength(0);
+ expect(drainedRequests).toHaveLength(0);
// Depth-limited requests should not reach the user's transformRequestFunction
expect(transformRequestFunction).not.toHaveBeenCalled();
@@ -2635,15 +2631,15 @@ describe('BasicCrawler', () => {
return;
}
- await context.enqueueLinks({
- urls: [
+ await context.addRequests(
+ [
'http://example.com/yes',
'http://example.com/no',
'http://example.com/no-globally',
'http://example.com/my-crawler/anything',
],
- label: 'child',
- });
+ { label: 'child' },
+ );
},
});
@@ -2670,7 +2666,7 @@ describe('BasicCrawler', () => {
respectRobotsTxtFile: { userAgent: 'MyCrawler' },
requestHandler: async (context) => {
if (context.request.label) return;
- await context.enqueueLinks({ urls: ['http://example.com/child'], label: 'child' });
+ await context.addRequests(['http://example.com/child'], { label: 'child' });
},
});
@@ -2706,7 +2702,7 @@ describe('BasicCrawler', () => {
crawler.stats.state.requestsFinished = 2;
- await context.enqueueLinks({ urls: requestsToAdd, label: 'not-undefined' });
+ await context.addRequests(requestsToAdd, { label: 'not-undefined' });
},
});
@@ -2735,7 +2731,7 @@ describe('BasicCrawler', () => {
return;
}
- await context.enqueueLinks({ urls: requestsToAdd, limit: 2, label: 'child' });
+ await context.addRequests(requestsToAdd, { limit: 2, label: 'child' });
},
});
@@ -2762,7 +2758,7 @@ describe('BasicCrawler', () => {
return;
}
- await context.enqueueLinks({ urls: requestsToAdd, limit: 1, label: 'child' });
+ await context.addRequests(requestsToAdd, { limit: 1, label: 'child' });
},
});
@@ -2794,7 +2790,7 @@ describe('BasicCrawler', () => {
return;
}
- await context.enqueueLinks({ urls: requestsToAdd, limit: 1, label: 'child' });
+ await context.addRequests(requestsToAdd, { limit: 1, label: 'child' });
},
});
@@ -2827,8 +2823,7 @@ describe('BasicCrawler', () => {
// This will add requests at depth+1, so initial requests add at depth 1 (allowed)
// and depth 1 requests add at depth 2 (blocked by maxCrawlDepth)
- await context.enqueueLinks({
- urls: requestsToAdd,
+ await context.addRequests(requestsToAdd, {
label: `depth-${context.request.crawlDepth + 1}`,
});
},
@@ -2932,7 +2927,7 @@ describe('BasicCrawler', () => {
// Enqueue 10 duplicate links + 1 new unique link
const urls = [...Array.from({ length: 10 }, () => 'http://example.com/'), 'http://example.com/new'];
- await context.enqueueLinks({ urls, label: 'child' });
+ await context.addRequests(urls, { label: 'child' });
},
});
@@ -3623,13 +3618,12 @@ describe('BasicCrawler', () => {
expect(await queue.isEmpty()).toBe(false);
});
- test('context.enqueueLinks validates userData against the label schema', async () => {
+ test('context.addRequests validates userData against the label schema', async () => {
const router = Router.create({ DETAIL: z.object({ id: z.string() }) });
let caught: unknown;
- router.addDefaultHandler(async ({ enqueueLinks }) => {
+ router.addDefaultHandler(async ({ addRequests }) => {
try {
- await enqueueLinks({
- urls: ['https://example.com/x'],
+ await addRequests(['https://example.com/x'], {
label: 'DETAIL',
userData: { id: 123 },
} as never);
diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.ts
index c7f3a1fa2..448bc6210 100644
--- a/test/core/enqueue_links/enqueue_links.test.ts
+++ b/test/core/enqueue_links/enqueue_links.test.ts
@@ -1,25 +1,20 @@
import {
- type AddRequestsBatchedOptions,
- cheerioCrawlerEnqueueLinks,
- MemoryStorageBackend,
- serviceLocator,
-} from '@crawlee/cheerio';
-import { enqueueLinks } from '@crawlee/core';
-import { launchPlaywright } from '@crawlee/playwright';
-import type { RequestQueueOperationOptions, Source } from '@crawlee/puppeteer';
-import {
- browserCrawlerEnqueueLinks,
+ CheerioCrawler,
EnqueueStrategy,
- launchPuppeteer,
+ MemoryStorageBackend,
Request,
RequestQueue,
-} from '@crawlee/puppeteer';
-import { type CheerioRoot } from '@crawlee/utils/internal';
+ serviceLocator,
+} from '@crawlee/cheerio';
+import type {
+ AddRequestsBatchedOptions,
+ EnqueueLinksOptions,
+ RequestQueueOperationOptions,
+ Source,
+} from '@crawlee/cheerio';
+import { BaseHttpClient, ResponseWithUrl } from '@crawlee/http-client';
+import { PlaywrightCrawler } from '@crawlee/playwright';
import { RobotsTxtFile } from '@crawlee/utils';
-import { load } from 'cheerio';
-import type { Browser as PlaywrightBrowser, Page as PlaywrightPage } from 'playwright';
-// @ts-ignore This only throws when compiled against puppeteer 25+ (ESM only), we only import types, so its alllll gooooood
-import type { Browser as PuppeteerBrowser, Page as PuppeteerPage } from 'puppeteer';
import log from '@apify/log';
@@ -50,23 +45,90 @@ const HTML = `
</html>
`;
-async function createRequestQueueMock() {
+/**
+ * A real `RequestQueue` (backed by the in-memory storage), with `addRequests` wrapped to also record
+ * every request it sees (other than the seed URL used to kick off the crawl) - so the queue still works
+ * for the crawler (`fetchNextRequest`, `isFinished`, ...), while tests get a flat list of everything that
+ * `enqueueLinks()` added.
+ */
+async function createRequestQueueMock(seedUrl = 'https://example.com') {
const enqueued: Source[] = [];
const requestQueue = await RequestQueue.open({ id: 'xxx' });
+ const originalAddRequests = requestQueue.addRequests.bind(requestQueue);
- // @ts-expect-error Override method for testing
- requestQueue.addRequests = async function (requests) {
- const processedRequests: Source[] = [];
+ requestQueue.addRequests = async (requests, options) => {
+ const items: Source[] = [];
for await (const request of requests) {
- processedRequests.push(typeof request === 'string' ? { url: request } : request);
+ items.push(typeof request === 'string' ? { url: request } : (request as Source));
}
- enqueued.push(...processedRequests);
- return { processedRequests, unprocessedRequests: [] as never[] };
+ enqueued.push(...items.filter((item) => item.url !== seedUrl));
+ return originalAddRequests(items, options);
};
return { enqueued, requestQueue };
}
+// Serves fixed HTML for every request, reporting the request's own URL back as the loaded URL so
+// `enqueueLinks()`'s hostname/domain-based strategies see whatever start URL a test navigates to,
+// regardless of where the content is actually served from.
+class FixtureHttpClient extends BaseHttpClient {
+ constructor(private readonly html: string) {
+ super();
+ }
+
+ protected async fetch(): Promise<Response> {
+ throw new Error('FixtureHttpClient.fetch() should never be called - sendRequest() is overridden directly.');
+ }
+
+ override async sendRequest(request: { url: string | URL }): Promise<Response> {
+ return new ResponseWithUrl(this.html, {
+ url: request.url.toString(),
+ status: 200,
+ headers: { 'content-type': 'text/html; charset=utf-8' },
+ });
+ }
+}
+
+const fixtureHttpClient = new FixtureHttpClient(HTML);
+
+async function runCheerioEnqueueLinks(
+ enqueueOptions: EnqueueLinksOptions,
+ { requestManager, startUrl = 'https://example.com' }: { requestManager: RequestQueue; startUrl?: string },
+) {
+ const crawler = new CheerioCrawler({
+ requestManager,
+ httpClient: fixtureHttpClient,
+ // Only the seed page extracts links - visited children would otherwise recursively re-run
+ // `enqueueLinks()` against the very same fixture page.
+ requestHandler: async ({ request, enqueueLinks }) => {
+ if (request.url !== startUrl) return;
+ await enqueueLinks(enqueueOptions);
+ },
+ });
+
+ await crawler.run([startUrl]);
+}
+
+async function runPlaywrightEnqueueLinks(
+ enqueueOptions: EnqueueLinksOptions,
+ { requestManager, startUrl = 'https://example.com' }: { requestManager: RequestQueue; startUrl?: string },
+) {
+ const crawler = new PlaywrightCrawler({
+ requestManager,
+ preNavigationHooks: [
+ async ({ page }) => {
+ await page.route('**/*', (route) => route.fulfill({ body: HTML, contentType: 'text/html' }));
+ },
+ ],
+ requestHandler: async ({ request, enqueueLinks }) => {
+ if (request.url !== startUrl) return;
+ await enqueueLinks(enqueueOptions);
+ },
+ });
+
+ await crawler.run([startUrl]);
+}
+
describe('enqueueLinks()', () => {
let ll: number;
beforeAll(() => {
@@ -82,416 +144,64 @@ describe('enqueueLinks()', () => {
log.setLevel(ll);
});
- describe.each([[launchPuppeteer], [launchPlaywright]] as const)('using %s', (method) => {
- let browser: PuppeteerBrowser | PlaywrightBrowser;
- let page: PuppeteerPage | PlaywrightPage;
-
- beforeEach(async () => {
- browser = (await method({ launchOptions: { headless: true } })) as PlaywrightBrowser | PuppeteerBrowser;
- page = await browser.newPage();
- await page.setContent(HTML);
- });
-
- afterEach(async () => {
- if (browser) await browser.close();
- page = null!;
- browser = null!;
- });
-
+ describe('using Playwright', () => {
test('works with item limit', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await browserCrawlerEnqueueLinks({
- options: { limit: 3, selector: '.click', strategy: EnqueueStrategy.All },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runPlaywrightEnqueueLinks(
+ { limit: 3, selector: '.click', strategy: EnqueueStrategy.All },
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('https://another.com/a/fifth');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
-
- expect(enqueued[3]).toBe(undefined);
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'https://example.com/a/b/first',
+ 'https://example.com/a/b/third',
+ 'https://another.com/a/fifth',
+ ]);
});
test('works with include (globs)', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- label: 'COOL',
- include,
- transformRequestFunction: (request) => {
- if (/example\.com\/a\/b\/third/.exec(request.url)) {
- request.method = 'OPTIONS';
- }
- return request;
- },
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({ label: 'COOL' });
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('OPTIONS');
- expect(enqueued[1].userData).toEqual({ label: 'COOL' });
-
- expect(enqueued[2].url).toBe('http://cool.com/');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({ label: 'COOL' });
- });
-
- test('does not throw with empty include patterns', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- const include = [
- 'https://example.com/**/*',
- '',
- { glob: ' ' },
- // Empty string used to throw an error (https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/issues/Wd0Ahfk9Vd2OPk4Uf)
- { glob: '' },
- '?(http|https)://cool.com/',
- ];
-
- await expect(
- browserCrawlerEnqueueLinks({
- options: { selector: '.click', include, strategy: EnqueueStrategy.All },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- }),
- ).resolves.not.toThrow();
+ await runPlaywrightEnqueueLinks(
+ { selector: '.click', strategy: EnqueueStrategy.All, label: 'COOL', include },
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
- });
-
- test('works with include (regexps)', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- const include = [/^https:\/\/example\.com\/(\w|\/)+/, /^(http|https):\/\/cool\.com\//];
-
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- include,
- transformRequestFunction: (request) => {
- if (/example\.com\/a\/b\/third/.exec(request.url)) {
- request.method = 'OPTIONS';
- }
- return request;
- },
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('OPTIONS');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('http://cool.com/');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
- });
-
- test('works with skipNavigation', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
-
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- skipNavigation: true,
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(2);
-
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'https://example.com/a/b/first',
+ 'https://example.com/a/b/third',
+ 'http://cool.com/',
+ ]);
for (const request of enqueued) {
- expect(request.skipNavigation).toBe(true);
+ expect(request.userData).toEqual({ label: 'COOL' });
}
});
- test('works with exclude glob', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
-
- const exclude = ['**/first'];
-
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- label: 'COOL',
- include,
- exclude,
- transformRequestFunction: (request) => {
- if (/example\.com\/a\/b\/third/.exec(request.url)) {
- request.method = 'OPTIONS';
- }
- return request;
- },
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(2);
-
- expect(enqueued[0].url).not.toBe('https://example.com/a/b/first');
- expect(enqueued[1].url).not.toBe('https://example.com/a/b/first');
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/third');
- expect(enqueued[0].method).toBe('OPTIONS');
- expect(enqueued[0].userData).toEqual({ label: 'COOL' });
-
- expect(enqueued[1].url).toBe('http://cool.com/');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({ label: 'COOL' });
- });
-
- test('works with exclude regexp', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
-
- const exclude = [/first/];
-
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- label: 'COOL',
- include,
- exclude,
- transformRequestFunction: (request) => {
- if (/example\.com\/a\/b\/third/.exec(request.url)) {
- request.method = 'OPTIONS';
- }
- return request;
- },
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(2);
-
- expect(enqueued[0].url).not.toBe('https://example.com/a/b/first');
- expect(enqueued[1].url).not.toBe('https://example.com/a/b/first');
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/third');
- expect(enqueued[0].method).toBe('OPTIONS');
- expect(enqueued[0].userData).toEqual({ label: 'COOL' });
-
- expect(enqueued[1].url).toBe('http://cool.com/');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({ label: 'COOL' });
- });
-
- test('works with no include/exclude filters (enqueues all matching strategy)', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
-
- await browserCrawlerEnqueueLinks({
- options: { selector: '.click', strategy: EnqueueStrategy.All },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(4);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('https://another.com/a/fifth');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
-
- expect(enqueued[3].url).toBe('http://cool.com/');
- expect(enqueued[3].method).toBe('GET');
- expect(enqueued[3].userData).toEqual({});
- });
-
- test('correctly resolves relative URLs with default strategy of same-hostname', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- await browserCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/' },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(2);
-
- expect(enqueued[0].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
- });
-
test('correctly resolves relative URLs with the strategy of same-domain', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await browserCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('http://example.absolute.com/hello');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
- });
-
- test('correctly resolves relative URLs with the strategy of all', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- await browserCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(8);
+ await runPlaywrightEnqueueLinks(
+ { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
+ { requestManager: requestQueue },
+ );
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/second');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('https://example.com/a/b/third');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
-
- expect(enqueued[3].url).toBe('https://another.com/a/fifth');
- expect(enqueued[3].method).toBe('GET');
- expect(enqueued[3].userData).toEqual({});
-
- expect(enqueued[4].url).toBe('http://cool.com/');
- expect(enqueued[4].method).toBe('GET');
- expect(enqueued[4].userData).toEqual({});
-
- expect(enqueued[5].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[5].method).toBe('GET');
- expect(enqueued[5].userData).toEqual({});
-
- expect(enqueued[6].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[6].method).toBe('GET');
- expect(enqueued[6].userData).toEqual({});
-
- expect(enqueued[7].url).toBe('http://example.absolute.com/hello');
- expect(enqueued[7].method).toBe('GET');
- expect(enqueued[7].userData).toEqual({});
- });
-
- test('correctly works with transformRequestFunction', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
-
- const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
-
- await browserCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- include,
- transformRequestFunction: (request) => {
- if (request.url.includes('example.com')) {
- request.method = 'POST';
- } else if (request.url.includes('cool.com')) {
- request.userData!.foo = 'bar';
- }
- return request;
- },
- },
- page,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('POST');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('POST');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('http://cool.com/');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData!.foo).toBe('bar');
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'http://www.absolute.com/x/absolutepath',
+ 'http://www.absolute.com/removethis/y/relativepath',
+ 'http://example.absolute.com/hello',
+ ]);
});
});
describe('using Cheerio', () => {
- let $: CheerioRoot;
-
- beforeEach(async () => {
- $ = load(HTML);
- });
-
- afterEach(async () => {
- $ = null!;
- });
-
test('works with include (globs)', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
strategy: EnqueueStrategy.All,
include,
@@ -502,10 +212,8 @@ describe('enqueueLinks()', () => {
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
@@ -524,15 +232,20 @@ describe('enqueueLinks()', () => {
test('does not throw with empty include patterns', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- const include = ['https://example.com/**/*', '?(http|https)://cool.com/', '', { glob: ' ' }];
+ const include = [
+ 'https://example.com/**/*',
+ '',
+ { glob: ' ' },
+ // Empty string used to throw an error (https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/issues/Wd0Ahfk9Vd2OPk4Uf)
+ { glob: '' },
+ '?(http|https)://cool.com/',
+ ];
await expect(
- cheerioCrawlerEnqueueLinks({
- options: { selector: '.click', include, strategy: EnqueueStrategy.All },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- }),
+ runCheerioEnqueueLinks(
+ { selector: '.click', include, strategy: EnqueueStrategy.All },
+ { requestManager: requestQueue },
+ ),
).resolves.not.toThrow();
expect(enqueued).toHaveLength(3);
@@ -542,8 +255,8 @@ describe('enqueueLinks()', () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const include = [/^https:\/\/example\.com\/(\w|\/)+/, /^(http|https):\/\/cool\.com\//];
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
strategy: EnqueueStrategy.All,
include,
@@ -554,43 +267,31 @@ describe('enqueueLinks()', () => {
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
expect(enqueued[1].url).toBe('https://example.com/a/b/third');
expect(enqueued[1].method).toBe('OPTIONS');
- expect(enqueued[1].userData).toEqual({});
expect(enqueued[2].url).toBe('http://cool.com/');
expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
});
test('works with include (mixed globs and regexps)', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const include = ['https://example.com/**/*', /^(http|https):\/\/cool\.com\//];
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- strategy: EnqueueStrategy.All,
- include,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks(
+ { selector: '.click', strategy: EnqueueStrategy.All, include },
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
-
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
expect(enqueued[1].url).toBe('https://example.com/a/b/third');
expect(enqueued[2].url).toBe('http://cool.com/');
@@ -603,12 +304,7 @@ describe('enqueueLinks()', () => {
// applies, so the cross-hostname URL is filtered out (aligned with crawlee-python).
const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
- await cheerioCrawlerEnqueueLinks({
- options: { selector: '.click', include },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks({ selector: '.click', include }, { requestManager: requestQueue });
expect(enqueued).toHaveLength(2);
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
@@ -617,213 +313,133 @@ describe('enqueueLinks()', () => {
test('throws when include is an empty array', async () => {
const { requestQueue } = await createRequestQueueMock();
+ let caughtError: unknown;
- await expect(
- cheerioCrawlerEnqueueLinks({
- options: { selector: '.click', include: [] },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- }),
- ).rejects.toThrow(/at `include`/);
- });
-
- test('works with no include/exclude filters (enqueues all matching strategy)', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { selector: '.click', strategy: EnqueueStrategy.All },
- $,
+ const crawler = new CheerioCrawler({
requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
+ httpClient: fixtureHttpClient,
+ requestHandler: async ({ enqueueLinks }) => {
+ try {
+ await enqueueLinks({ selector: '.click', include: [] });
+ } catch (error) {
+ caughtError = error;
+ }
+ },
});
- expect(enqueued).toHaveLength(4);
+ await crawler.run(['https://example.com']);
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('https://example.com/a/b/third');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
+ expect(caughtError).toBeInstanceOf(Error);
+ expect((caughtError as Error).message).toMatch(/at `include`/);
+ });
- expect(enqueued[2].url).toBe('https://another.com/a/fifth');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
+ test('works with no include/exclude filters (enqueues all matching strategy)', async () => {
+ const { enqueued, requestQueue } = await createRequestQueueMock();
+ await runCheerioEnqueueLinks(
+ { selector: '.click', strategy: EnqueueStrategy.All },
+ { requestManager: requestQueue },
+ );
- expect(enqueued[3].url).toBe('http://cool.com/');
- expect(enqueued[3].method).toBe('GET');
- expect(enqueued[3].userData).toEqual({});
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'https://example.com/a/b/first',
+ 'https://example.com/a/b/third',
+ 'https://another.com/a/fifth',
+ 'http://cool.com/',
+ ]);
});
- test('correctly resolves relative URLs with the strategy of all', async () => {
+ test('works with exclude glob', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(8);
+ const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
+ const exclude = ['**/first'];
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
+ await runCheerioEnqueueLinks(
+ { selector: '.click', strategy: EnqueueStrategy.All, include, exclude },
+ { requestManager: requestQueue },
+ );
- expect(enqueued[1].url).toBe('https://example.com/a/second');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
+ expect(enqueued.map((r) => r.url)).toEqual(['https://example.com/a/b/third', 'http://cool.com/']);
+ });
- expect(enqueued[2].url).toBe('https://example.com/a/b/third');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
+ test('works with exclude regexp', async () => {
+ const { enqueued, requestQueue } = await createRequestQueueMock();
+ const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
+ const exclude = [/first/];
- expect(enqueued[3].url).toBe('https://another.com/a/fifth');
- expect(enqueued[3].method).toBe('GET');
- expect(enqueued[3].userData).toEqual({});
+ await runCheerioEnqueueLinks(
+ { selector: '.click', strategy: EnqueueStrategy.All, include, exclude },
+ { requestManager: requestQueue },
+ );
- expect(enqueued[4].url).toBe('http://cool.com/');
- expect(enqueued[4].method).toBe('GET');
- expect(enqueued[4].userData).toEqual({});
+ expect(enqueued.map((r) => r.url)).toEqual(['https://example.com/a/b/third', 'http://cool.com/']);
+ });
- expect(enqueued[5].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[5].method).toBe('GET');
- expect(enqueued[5].userData).toEqual({});
+ test('works with skipNavigation', async () => {
+ const { enqueued, requestQueue } = await createRequestQueueMock();
- expect(enqueued[6].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[6].method).toBe('GET');
- expect(enqueued[6].userData).toEqual({});
+ await runCheerioEnqueueLinks(
+ { selector: '.click', skipNavigation: true },
+ { requestManager: requestQueue },
+ );
- expect(enqueued[7].url).toBe('http://example.absolute.com/hello');
- expect(enqueued[7].method).toBe('GET');
- expect(enqueued[7].userData).toEqual({});
+ expect(enqueued).toHaveLength(2);
+ for (const request of enqueued) {
+ expect(request.skipNavigation).toBe(true);
+ }
});
test('correctly resolves relative URLs with the default strategy of same-hostname', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/' },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks(
+ { baseUrl: 'http://www.absolute.com/removethis/' },
+ { requestManager: requestQueue },
+ );
- expect(enqueued).toHaveLength(2);
-
- expect(enqueued[0].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'http://www.absolute.com/x/absolutepath',
+ 'http://www.absolute.com/removethis/y/relativepath',
+ ]);
});
test('correctly resolves relative URLs with the strategy of same-domain', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks(
+ { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.SameDomain },
+ { requestManager: requestQueue },
+ );
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('http://www.absolute.com/x/absolutepath');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('http://www.absolute.com/removethis/y/relativepath');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
-
- expect(enqueued[2].url).toBe('http://example.absolute.com/hello');
- expect(enqueued[2].method).toBe('GET');
- expect(enqueued[2].userData).toEqual({});
+ expect(enqueued.map((r) => r.url)).toEqual([
+ 'http://www.absolute.com/x/absolutepath',
+ 'http://www.absolute.com/removethis/y/relativepath',
+ 'http://example.absolute.com/hello',
+ ]);
});
- test('ignores an explicitly undefined baseUrl and keeps the resolved one', async () => {
+ test('correctly resolves relative URLs with the strategy of all', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { strategy: EnqueueStrategy.SameDomain, baseUrl: undefined },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks(
+ { baseUrl: 'http://www.absolute.com/removethis/', strategy: EnqueueStrategy.All },
+ { requestManager: requestQueue },
+ );
- expect(enqueued.map((request) => request.url)).toEqual([
+ expect(enqueued.map((r) => r.url)).toEqual([
'https://example.com/a/b/first',
'https://example.com/a/second',
'https://example.com/a/b/third',
- 'https://example.com/x/absolutepath',
- 'https://example.com/y/relativepath',
+ 'https://another.com/a/fifth',
+ 'http://cool.com/',
+ 'http://www.absolute.com/x/absolutepath',
+ 'http://www.absolute.com/removethis/y/relativepath',
+ 'http://example.absolute.com/hello',
]);
});
- test('ignores an explicitly undefined baseUrl when delegating to the bound enqueueLinks', async () => {
- let forwardedBaseUrl: string | undefined;
-
- await cheerioCrawlerEnqueueLinks({
- options: { strategy: EnqueueStrategy.SameDomain, baseUrl: undefined },
- $,
- originalRequestUrl: 'https://example.com',
- enqueueLinks: async (options?: { baseUrl?: string }) => {
- forwardedBaseUrl = options?.baseUrl;
- return { processedRequests: [], unprocessedRequests: [] };
- },
- });
-
- expect(forwardedBaseUrl).toBe('https://example.com');
- });
-
- test('keeps filtering by the original domain with the strategy of same-domain after an off-domain redirect', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: { strategy: EnqueueStrategy.SameDomain },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- finalRequestUrl: 'https://another.com/',
- });
-
- expect(enqueued).toHaveLength(3);
-
- expect(enqueued[0].url).toBe('https://example.com/a/b/first');
- expect(enqueued[1].url).toBe('https://example.com/a/second');
- expect(enqueued[2].url).toBe('https://example.com/a/b/third');
- });
-
- test('correctly resolves relative URLs with `urls` option', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: {
- baseUrl: 'http://www.absolute.com/removethis/',
- urls: ['/relative/url1', '/relative/url2'],
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
-
- expect(enqueued).toHaveLength(2);
-
- expect(enqueued[0].url).toBe('http://www.absolute.com/relative/url1');
- expect(enqueued[0].method).toBe('GET');
- expect(enqueued[0].userData).toEqual({});
-
- expect(enqueued[1].url).toBe('http://www.absolute.com/relative/url2');
- expect(enqueued[1].method).toBe('GET');
- expect(enqueued[1].userData).toEqual({});
- });
-
test('correctly works with transformRequestFunction', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const include = ['https://example.com/**/*', '?(http|https)://cool.com/'];
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
strategy: EnqueueStrategy.All,
include,
@@ -836,20 +452,16 @@ describe('enqueueLinks()', () => {
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
expect(enqueued[0].method).toBe('POST');
- expect(enqueued[0].userData).toEqual({});
expect(enqueued[1].url).toBe('https://example.com/a/b/third');
expect(enqueued[1].method).toBe('POST');
- expect(enqueued[1].userData).toEqual({});
expect(enqueued[2].url).toBe('http://cool.com/');
expect(enqueued[2].method).toBe('GET');
@@ -859,58 +471,46 @@ describe('enqueueLinks()', () => {
test('accepts forefront option', async () => {
const enqueued: { request: Source; options?: RequestQueueOperationOptions }[] = [];
const requestQueue = await RequestQueue.open({ id: 'xxx' });
+ const originalAddRequests = requestQueue.addRequests.bind(requestQueue);
requestQueue.addRequests = async (requests, options) => {
- // copy the requests to the enqueued list, along with options that were passed to addRequests,
- // so that it doesn't matter how many calls were made
+ const items: Source[] = [];
for await (const request of requests) {
- enqueued.push({ request: typeof request === 'string' ? { url: request } : request, options });
+ const source = typeof request === 'string' ? { url: request } : request;
+ items.push(source);
+ if (source.url !== 'https://example.com') enqueued.push({ request: source, options });
}
- return { processedRequests: [], unprocessedRequests: [] };
+ return originalAddRequests(items, options);
};
- await cheerioCrawlerEnqueueLinks({
- options: {
- forefront: true,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks({ forefront: true }, { requestManager: requestQueue });
expect(enqueued).toHaveLength(5);
-
- for (let i = 0; i < 5; i++) {
- expect(enqueued[i].options!.forefront).toBe(true);
+ for (const { options } of enqueued) {
+ expect(options!.forefront).toBe(true);
}
});
test('accepts waitForAllRequestsToBeAdded option', async () => {
const enqueued: { request: string | Source; options?: AddRequestsBatchedOptions }[] = [];
const requestQueue = await RequestQueue.open({ id: 'xxx' });
+ const originalAddRequestsBatched = requestQueue.addRequestsBatched.bind(requestQueue);
requestQueue.addRequestsBatched = async (requests, options) => {
- // copy the requests to the enqueued list, along with options that were passed to addRequests,
- // so that it doesn't matter how many calls were made
+ const items: Source[] = [];
for await (const request of requests) {
- enqueued.push({ request: typeof request === 'string' ? { url: request } : request, options });
+ const source = typeof request === 'string' ? { url: request } : request;
+ items.push(source);
+ if (source.url !== 'https://example.com') enqueued.push({ request: source, options });
}
- return { addedRequests: [], waitForAllRequestsToBeAdded: Promise.resolve([]), requestsOverLimit: [] };
+ return originalAddRequestsBatched(items, options);
};
- await cheerioCrawlerEnqueueLinks({
- options: {
- waitForAllRequestsToBeAdded: true,
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks({ waitForAllRequestsToBeAdded: true }, { requestManager: requestQueue });
expect(enqueued).toHaveLength(5);
-
- for (let i = 0; i < 5; i++) {
- expect(enqueued[i].options!.waitForAllRequestsToBeAdded).toBe(true);
+ for (const { options } of enqueued) {
+ expect(options!.waitForAllRequestsToBeAdded).toBe(true);
}
});
@@ -918,19 +518,12 @@ describe('enqueueLinks()', () => {
test('global label option is applied if no other label is provided', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: {
- selector: '.click',
- label: 'global-label',
- include: ['https://example.com/**/*'],
- },
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ await runCheerioEnqueueLinks(
+ { selector: '.click', label: 'global-label', include: ['https://example.com/**/*'] },
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(2);
- // Global label should be applied to all matched requests
expect(enqueued[0].userData).toEqual({ label: 'global-label' });
expect(enqueued[1].userData).toEqual({ label: 'global-label' });
});
@@ -938,8 +531,8 @@ describe('enqueueLinks()', () => {
test('transformRequestFunction overrides global label', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
label: 'global-label',
include: [/example\.com/],
@@ -950,16 +543,12 @@ describe('enqueueLinks()', () => {
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(2);
- // transformRequestFunction should override global label
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
expect(enqueued[0].userData).toEqual({ label: 'transformed-label' });
- // URL not modified by transformRequestFunction should keep global label
expect(enqueued[1].url).toBe('https://example.com/a/b/third');
expect(enqueued[1].userData).toEqual({ label: 'global-label' });
});
@@ -967,25 +556,21 @@ describe('enqueueLinks()', () => {
test('transformRequestFunction can override global label for all requests', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
strategy: EnqueueStrategy.All,
label: 'global-label',
include: ['https://example.com/a/b/first', 'https://example.com/a/b/third', 'http://cool.com/'],
transformRequestFunction: (request) => {
- // Override all labels
request.label = 'final-label';
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(3);
- // All requests should have the transformed label
for (const request of enqueued) {
expect(request.userData).toEqual({ label: 'final-label' });
}
@@ -994,27 +579,22 @@ describe('enqueueLinks()', () => {
test('transformRequestFunction can modify request properties', async () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
include: [/example\.com/],
transformRequestFunction: (request) => {
- // Set method and userData via transformRequestFunction
request.method = 'PUT';
request.userData = { ...request.userData, transformed: true };
return request;
},
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(2);
- // transformRequestFunction should set method
expect(enqueued[0].method).toBe('PUT');
expect(enqueued[1].method).toBe('PUT');
- // userData should contain transformed data
expect(enqueued[0].userData).toEqual({ transformed: true });
expect(enqueued[1].userData).toEqual({ transformed: true });
});
@@ -1022,43 +602,38 @@ describe('enqueueLinks()', () => {
test('transformRequestFunction can return a new plain object instead of modifying in place', async () => {
const enqueued: Source[] = [];
const requestQueue = await RequestQueue.open({ id: 'xxx' });
+ const originalAddRequestsBatched = requestQueue.addRequestsBatched.bind(requestQueue);
- // Custom mock that checks for Request instances - we override addRequestsBatched
- // to verify that request options returned by transformRequestFunction are converted to Request instances
- requestQueue.addRequestsBatched = async (requests) => {
- // @ts-expect-error Iterating over the requests parameter which has a narrower type in the override
- for (const request of requests) {
- // This check ensures that request options are converted to Request instances
+ // Custom mock that checks for Request instances - we wrap addRequestsBatched to verify that
+ // request options returned by transformRequestFunction are converted to Request instances
+ requestQueue.addRequestsBatched = async (requests, options) => {
+ const items: Request[] = [];
+ for await (const request of requests) {
if (!(request instanceof Request)) {
throw new Error(
`Expected Request instance but got plain object: ${JSON.stringify(request)}`,
);
}
- enqueued.push(request);
+ items.push(request);
+ if (request.url !== 'https://example.com') enqueued.push(request);
}
- return { addedRequests: [], waitForAllRequestsToBeAdded: Promise.resolve([]) };
+ return originalAddRequestsBatched(items, options);
};
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
include: ['https://example.com/**/*'],
- transformRequestFunction: (request) => {
- // Return a new plain object instead of modifying in place
- return {
- url: request.url,
- method: 'DELETE' as const,
- userData: { replaced: true },
- };
- },
+ transformRequestFunction: (request) => ({
+ url: request.url,
+ method: 'DELETE' as const,
+ userData: { replaced: true },
+ }),
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(2);
- // The request options should be properly converted to a Request
expect(enqueued[0].url).toBe('https://example.com/a/b/first');
expect(enqueued[0].method).toBe('DELETE');
expect(enqueued[0].userData).toEqual({ replaced: true });
@@ -1071,8 +646,8 @@ describe('enqueueLinks()', () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const onSkippedRequest = vi.fn();
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
label: 'global-label',
include: ['https://example.com/**/*'],
@@ -1080,32 +655,23 @@ describe('enqueueLinks()', () => {
if (request.url.includes('/a/b/first')) {
return 'skip';
}
- // 'unchanged' should keep the original options (including global label)
return 'unchanged';
},
onSkippedRequest,
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
- // 'skip' should exclude /a/b/first, 'unchanged' should keep /a/b/third as-is
expect(enqueued).toHaveLength(1);
expect(enqueued[0].url).toBe('https://example.com/a/b/third');
expect(enqueued[0].userData).toEqual({ label: 'global-label' });
- // 'skip' should trigger onSkippedRequest with reason 'transform'
const skippedCalls = onSkippedRequest.mock.calls.map(
(call: unknown[]) => call[0] as { url: string; reason: string },
);
const transformSkipped = skippedCalls.filter((s) => s.url === 'https://example.com/a/b/first');
expect(transformSkipped).toHaveLength(1);
- expect(transformSkipped[0]).toEqual({
- url: 'https://example.com/a/b/first',
- reason: 'transform',
- });
- // 'unchanged' should NOT trigger onSkippedRequest
+ expect(transformSkipped[0]).toEqual({ url: 'https://example.com/a/b/first', reason: 'transform' });
const unchangedSkipped = skippedCalls.filter((s) => s.url === 'https://example.com/a/b/third');
expect(unchangedSkipped).toHaveLength(0);
});
@@ -1114,12 +680,11 @@ describe('enqueueLinks()', () => {
const { enqueued, requestQueue } = await createRequestQueueMock();
const onSkippedRequest = vi.fn();
- await cheerioCrawlerEnqueueLinks({
- options: {
+ await runCheerioEnqueueLinks(
+ {
selector: '.click',
include: ['https://example.com/**/*'],
transformRequestFunction: (request) => {
- // Skip the first URL, keep the second
if (request.url.includes('/a/b/first')) {
return false;
}
@@ -1127,25 +692,18 @@ describe('enqueueLinks()', () => {
},
onSkippedRequest,
},
- $,
- requestManager: requestQueue,
- originalRequestUrl: 'https://example.com',
- });
+ { requestManager: requestQueue },
+ );
expect(enqueued).toHaveLength(1);
expect(enqueued[0].url).toBe('https://example.com/a/b/third');
- // onSkippedRequest fires for URLs filtered out by include (another.com, cool.com)
- // AND for the URL explicitly skipped by transformRequestFunction
const skippedCalls = onSkippedRequest.mock.calls.map(
(call: unknown[]) => call[0] as { url: string; reason: string },
);
const transformSkipped = skippedCalls.filter((s) => s.url === 'https://example.com/a/b/first');
expect(transformSkipped).toHaveLength(1);
- expect(transformSkipped[0]).toEqual({
- url: 'https://example.com/a/b/first',
- reason: 'transform',
- });
+ expect(transformSkipped[0]).toEqual({ url: 'https://example.com/a/b/first', reason: 'transform' });
});
});
});
@@ -1170,24 +728,38 @@ describe('enqueueLinks()', () => {
'http://example.com/my-crawler/anything',
];
- test('defaults to the catch-all user-agent when not provided', async () => {
+ const robotsHtml = `<html><body>${urls.map((url) => `<a href="${url}">link</a>`).join('')}</body></html>`;
+ const robotsHttpClient = new FixtureHttpClient(robotsHtml);
+
+ async function runWithRobotsTxt(respectRobotsTxtFile?: boolean | { userAgent?: string }) {
const { enqueued, requestQueue } = await createRequestQueueMock();
- await enqueueLinks({ urls, requestManager: requestQueue, robotsTxtFile });
+ const crawler = new (class extends CheerioCrawler {
+ override async getRobotsTxtFileForUrl(_url: string) {
+ return robotsTxtFile;
+ }
+ })({
+ requestManager: requestQueue,
+ httpClient: robotsHttpClient,
+ respectRobotsTxtFile: respectRobotsTxtFile ?? true,
+ requestHandler: async ({ request, enqueueLinks }) => {
+ if (request.url !== 'https://example.com') return;
+ await enqueueLinks({ strategy: EnqueueStrategy.All });
+ },
+ });
+
+ await crawler.run(['https://example.com']);
+
+ return enqueued;
+ }
+ test('defaults to the catch-all user-agent when not provided', async () => {
+ const enqueued = await runWithRobotsTxt();
expect(enqueued.map((r) => r.url)).toEqual(['http://example.com/yes']);
});
test('applies rules for the configured user-agent', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
-
- await enqueueLinks({
- urls,
- requestManager: requestQueue,
- robotsTxtFile,
- respectRobotsTxtFile: { userAgent: 'MyCrawler' },
- });
-
+ const enqueued = await runWithRobotsTxt({ userAgent: 'MyCrawler' });
expect(enqueued.map((r) => r.url)).toEqual([
'http://example.com/yes',
'http://example.com/my-crawler/anything',
@@ -1195,15 +767,7 @@ describe('enqueueLinks()', () => {
});
test('skips filtering when set to false even if robotsTxtFile is provided', async () => {
- const { enqueued, requestQueue } = await createRequestQueueMock();
-
- await enqueueLinks({
- urls,
- requestManager: requestQueue,
- robotsTxtFile,
- respectRobotsTxtFile: false,
- });
-
+ const enqueued = await runWithRobotsTxt(false);
expect(enqueued.map((r) => r.url)).toEqual(urls);
});
});
diff --git a/test/core/router.test.ts b/test/core/router.test.ts
index 8d1b4b49f..89880826d 100644
--- a/test/core/router.test.ts
+++ b/test/core/router.test.ts
@@ -450,7 +450,7 @@ describe('Router', () => {
router.addHandler('PRODUCT', async ({ addRequests, enqueueLinks }) => {
// context methods are typed from the route map
await addRequests([{ url: 'https://e.com/c', label: 'CATEGORY', userData: { categoryId: 'c1' } }]);
- await enqueueLinks({ urls: ['https://e.com/p'], label: 'PRODUCT', userData: { sku: 's', price: 1 } });
+ await enqueueLinks({ label: 'PRODUCT', userData: { sku: 's', price: 1 } });
// @ts-expect-error wrong userData shape for the label
await addRequests([{ url: 'https://e.com/p', label: 'PRODUCT', userData: { categoryId: 'x' } }]);
// @ts-expect-error label not present in the route map