Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: use `@apilink` + render package readmes on overview page

Martin Adámek61e396452332
diff --git a/packages/basic-crawler/README.md b/packages/basic-crawler/README.md
new file mode 100644
index 000000000..032df77c5
--- /dev/null
+++ b/packages/basic-crawler/README.md
@@ -0,0 +1,44 @@
+# `@crawlee/basic`
+
+Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+`BasicCrawler` is a low-level tool that requires the user to implement the page download and data extraction functionality themselves.
+If we want a crawler that already facilitates this functionality, we should consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler).
+
+`BasicCrawler` invokes the user-provided [`requestHandler`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) for each [Request](https://crawlee.dev/api/core/class/Request) object, which represents a single URL to crawl. The [Request](https://crawlee.dev/api/core/class/Request) objects are fed from the [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/api/basic-crawler/class/BasicCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/api/basic-crawler/class/BasicCrawler#run) function is provided.
+
+If both [`requestList`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes if there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#autoscaledPoolOptions) parameter of the `BasicCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) constructor are available directly in the `BasicCrawler` constructor.
+
+## Example usage
+
+```javascript
+import { BasicCrawler, Dataset } from 'crawlee';
+
+// Create a crawler instance
+const crawler = new BasicCrawler({
+ async requestHandler({ request, sendRequest }) {
+ // 'request' contains an instance of the Request class
+ // Here we simply fetch the HTML of the page and store it to a dataset
+ const { body } = await sendRequest({
+ url: request.url,
+ method: request.method,
+ body: request.payload,
+ headers: request.headers,
+ });
+
+ await Dataset.pushData({
+ url: request.url,
+ html: body,
+ })
+ },
+});
+
+// Enqueue the initial requests and run the crawler
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+]);
diff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.ts
index af4a4cd2e..de349be5d 100644
--- a/packages/basic-crawler/src/internals/basic-crawler.ts
+++ b/packages/basic-crawler/src/internals/basic-crawler.ts
@@ -68,38 +68,38 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
/**
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
- * The function receives the {@link BasicCrawlingContext} as an argument,
- * where the {@link BasicCrawlingContext.request|`request`} represents the URL to crawl.
+ * The function receives the {@apilink BasicCrawlingContext} as an argument,
+ * where the {@apilink BasicCrawlingContext.request|`request`} represents the URL to crawl.
*
* The function must return a promise, which is then awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to the {@link BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * request later, up to the {@apilink BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
* If all the retries fail, the crawler calls the function
- * provided to the {@link BasicCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
+ * provided to the {@apilink BasicCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
* To make this work, we should **always**
* let our function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
+ * {@apilink Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
*/
requestHandler?: RequestHandler<Context>;
/**
* User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
- * The function receives the {@link BasicCrawlingContext} as an argument,
- * where the {@link BasicCrawlingContext.request|`request`} represents the URL to crawl.
+ * The function receives the {@apilink BasicCrawlingContext} as an argument,
+ * where the {@apilink BasicCrawlingContext.request|`request`} represents the URL to crawl.
*
* The function must return a promise, which is then awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to the {@link BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * request later, up to the {@apilink BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
* If all the retries fail, the crawler calls the function
- * provided to the {@link BasicCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
+ * provided to the {@apilink BasicCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
* To make this work, we should **always**
* let our function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
+ * {@apilink Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
*
* @deprecated `handleRequestFunction` has been renamed to `requestHandler` and will be removed in a future version.
* @ignore
@@ -108,28 +108,28 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
/**
* Static list of URLs to be processed.
- * If not provided, the crawler will open the default request queue when the {@link BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
- * > Alternatively, `requests` parameter of {@link BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
+ * If not provided, the crawler will open the default request queue when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
+ * > Alternatively, `requests` parameter of {@apilink BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
* it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`.
*/
requestList?: RequestList;
/**
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * If not provided, the crawler will open the default request queue when the {@link BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
- * > Alternatively, `requests` parameter of {@link BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
+ * If not provided, the crawler will open the default request queue when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called.
+ * > Alternatively, `requests` parameter of {@apilink BasicCrawler.run|`crawler.run()`} could be used to enqueue the initial requests -
* it is a shortcut for running `crawler.addRequests()` before the `crawler.run()`.
*/
requestQueue?: RequestQueue;
/**
- * Timeout in which the function passed as {@link BasicCrawlerOptions.requestHandler|`requestHandler`} needs to finish, in seconds.
+ * Timeout in which the function passed as {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`} needs to finish, in seconds.
* @default 60
*/
requestHandlerTimeoutSecs?: number;
/**
- * Timeout in which the function passed as {@link BasicCrawlerOptions.requestHandler|`requestHandler`} needs to finish, in seconds.
+ * Timeout in which the function passed as {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`} needs to finish, in seconds.
* @default 60
* @deprecated `handleRequestTimeoutSecs` has been renamed to `requestHandlerTimeoutSecs` and will be removed in a future version.
* @ignore
@@ -138,30 +138,30 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
/**
* User-provided function that allows modifying the request object before it gets retried by the crawler.
- * It's executed before each retry for the requests that failed less than {@link BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * It's executed before each retry for the requests that failed less than {@apilink BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
*
- * The function receives the {@link BasicCrawlingContext} as the first argument,
- * where the {@link BasicCrawlingContext.request|`request`} corresponds to the request to be retried.
+ * The function receives the {@apilink BasicCrawlingContext} as the first argument,
+ * where the {@apilink BasicCrawlingContext.request|`request`} corresponds to the request to be retried.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*/
errorHandler?: ErrorHandler<Context>;
/**
- * A function to handle requests that failed more than {@link BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * A function to handle requests that failed more than {@apilink BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
*
- * The function receives the {@link BasicCrawlingContext} as the first argument,
- * where the {@link BasicCrawlingContext.request|`request`} corresponds to the failed request.
+ * The function receives the {@apilink BasicCrawlingContext} as the first argument,
+ * where the {@apilink BasicCrawlingContext.request|`request`} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*/
failedRequestHandler?: ErrorHandler<Context>;
/**
- * A function to handle requests that failed more than {@link BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * A function to handle requests that failed more than {@apilink BasicCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
*
- * The function receives the {@link BasicCrawlingContext} as the first argument,
- * where the {@link BasicCrawlingContext.request|`request`} corresponds to the failed request.
+ * The function receives the {@apilink BasicCrawlingContext} as the first argument,
+ * where the {@apilink BasicCrawlingContext.request|`request`} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*
@@ -171,7 +171,7 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
handleFailedRequestFunction?: ErrorHandler<Context>;
/**
- * Indicates how many times the request is retried if {@link BasicCrawlerOptions.requestHandler|`requestHandler`} fails.
+ * Indicates how many times the request is retried if {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`} fails.
* @default 3
*/
maxRequestRetries?: number;
@@ -184,17 +184,17 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
maxRequestsPerCrawl?: number;
/**
- * Custom options passed to the underlying {@link AutoscaledPool} constructor.
- * > *NOTE:* The {@link AutoscaledPoolOptions.runTaskFunction|`runTaskFunction`}
- * and {@link AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`} options
+ * Custom options passed to the underlying {@apilink AutoscaledPool} constructor.
+ * > *NOTE:* The {@apilink AutoscaledPoolOptions.runTaskFunction|`runTaskFunction`}
+ * and {@apilink AutoscaledPoolOptions.isTaskReadyFunction|`isTaskReadyFunction`} options
* are provided by the crawler and cannot be overridden.
- * However, we can provide a custom implementation of {@link AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`}.
+ * However, we can provide a custom implementation of {@apilink AutoscaledPoolOptions.isFinishedFunction|`isFinishedFunction`}.
*/
autoscaledPoolOptions?: AutoscaledPoolOptions;
/**
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the
- * AutoscaledPool {@link AutoscaledPoolOptions.minConcurrency|`minConcurrency`} option.
+ * AutoscaledPool {@apilink AutoscaledPoolOptions.minConcurrency|`minConcurrency`} option.
* > *WARNING:* If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash.
* If not sure, it's better to keep the default value and the concurrency will scale up automatically.
*/
@@ -202,25 +202,25 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
/**
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the
- * AutoscaledPool {@link AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} option.
+ * AutoscaledPool {@apilink AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} option.
*/
maxConcurrency?: number;
/**
* The maximum number of requests per minute the crawler should run.
* By default, this is set to `Infinity`, but we can pass any positive, non-zero integer.
- * Shortcut for the AutoscaledPool {@link AutoscaledPoolOptions.maxTasksPerMinute|`maxTasksPerMinute`} option.
+ * Shortcut for the AutoscaledPool {@apilink AutoscaledPoolOptions.maxTasksPerMinute|`maxTasksPerMinute`} option.
*/
maxRequestsPerMinute?: number;
/**
- * Basic crawler will initialize the {@link SessionPool} with the corresponding {@link SessionPoolOptions|`sessionPoolOptions`}.
- * The session instance will be than available in the {@link BasicCrawlerOptions.requestHandler|`requestHandler`}.
+ * Basic crawler will initialize the {@apilink SessionPool} with the corresponding {@apilink SessionPoolOptions|`sessionPoolOptions`}.
+ * The session instance will be than available in the {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`}.
*/
useSessionPool?: boolean;
/**
- * The configuration options for {@link SessionPool} to use.
+ * The configuration options for {@apilink SessionPool} to use.
*/
sessionPoolOptions?: SessionPoolOptions;
@@ -236,29 +236,29 @@ export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCraw
* `BasicCrawler` is a low-level tool that requires the user to implement the page
* download and data extraction functionality themselves.
* If we want a crawler that already facilitates this functionality,
- * we should consider using {@link CheerioCrawler}, {@link PuppeteerCrawler} or {@link PlaywrightCrawler}.
+ * we should consider using {@apilink CheerioCrawler}, {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler}.
*
- * `BasicCrawler` invokes the user-provided {@link BasicCrawlerOptions.requestHandler|`requestHandler`}
- * for each {@link Request} object, which represents a single URL to crawl.
- * The {@link Request} objects are fed from the {@link RequestList} or {@link RequestQueue}
- * instances provided by the {@link BasicCrawlerOptions.requestList|`requestList`} or {@link BasicCrawlerOptions.requestQueue|`requestQueue`}
+ * `BasicCrawler` invokes the user-provided {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`}
+ * for each {@apilink Request} object, which represents a single URL to crawl.
+ * The {@apilink Request} objects are fed from the {@apilink RequestList} or {@apilink RequestQueue}
+ * instances provided by the {@apilink BasicCrawlerOptions.requestList|`requestList`} or {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`}
* constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided,
- * the crawler will open the default request queue either when the {@link BasicCrawler.addRequests|`crawler.addRequests()`} function is called,
- * or if `requests` parameter (representing the initial requests) of the {@link BasicCrawler.run|`crawler.run()`} function is provided.
+ * the crawler will open the default request queue either when the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} function is called,
+ * or if `requests` parameter (representing the initial requests) of the {@apilink BasicCrawler.run|`crawler.run()`} function is provided.
*
- * If both {@link BasicCrawlerOptions.requestList|`requestList`} and {@link BasicCrawlerOptions.requestQueue|`requestQueue`} options are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to the {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * If both {@apilink BasicCrawlerOptions.requestList|`requestList`} and {@apilink BasicCrawlerOptions.requestQueue|`requestQueue`} options are used,
+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
+ * to the {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
- * The crawler finishes if there are no more {@link Request} objects to crawl.
+ * The crawler finishes if there are no more {@apilink Request} objects to crawl.
*
* New requests are only dispatched when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the {@link BasicCrawlerOptions.autoscaledPoolOptions|`autoscaledPoolOptions`}
+ * using the functionality provided by the {@apilink AutoscaledPool} class.
+ * All {@apilink AutoscaledPool} configuration options can be passed to the {@apilink BasicCrawlerOptions.autoscaledPoolOptions|`autoscaledPoolOptions`}
* parameter of the `BasicCrawler` constructor.
- * For user convenience, the {@link AutoscaledPoolOptions.minConcurrency|`minConcurrency`} and
- * {@link AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} options of the
- * underlying {@link AutoscaledPool} constructor are available directly in the `BasicCrawler` constructor.
+ * For user convenience, the {@apilink AutoscaledPoolOptions.minConcurrency|`minConcurrency`} and
+ * {@apilink AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} options of the
+ * underlying {@apilink AutoscaledPool} constructor are available directly in the `BasicCrawler` constructor.
*
* **Example usage:**
*
@@ -296,41 +296,41 @@ export class BasicCrawler<Context extends CrawlingContext = BasicCrawlingContext
private static readonly CRAWLEE_STATE_KEY = 'CRAWLEE_STATE';
/**
- * A reference to the underlying {@link Statistics} class that collects and logs run statistics for requests.
+ * A reference to the underlying {@apilink Statistics} class that collects and logs run statistics for requests.
*/
readonly stats: Statistics;
/**
- * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request|requests}.
+ * A reference to the underlying {@apilink RequestList} class that manages the crawler's {@apilink Request|requests}.
* Only available if used by the crawler.
*/
requestList?: RequestList;
/**
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
- * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request|requests}.
+ * A reference to the underlying {@apilink RequestQueue} class that manages the crawler's {@apilink Request|requests}.
* Only available if used by the crawler.
*/
requestQueue?: RequestQueue;
/**
- * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session|sessions}.
+ * A reference to the underlying {@apilink SessionPool} class that manages the crawler's {@apilink Session|sessions}.
* Only available if used by the crawler.
*/
sessionPool?: SessionPool;
/**
- * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
- * > *NOTE:* This property is only initialized after calling the {@link BasicCrawler.run|`crawler.run()`} function.
+ * A reference to the underlying {@apilink AutoscaledPool} class that manages the concurrency of the crawler.
+ * > *NOTE:* This property is only initialized after calling the {@apilink BasicCrawler.run|`crawler.run()`} function.
* We can use it to change the concurrency settings on the fly,
- * to pause the crawler by calling {@link AutoscaledPool.pause|`autoscaledPool.pause()`}
- * or to abort it by calling {@link AutoscaledPool.abort|`autoscaledPool.abort()`}.
+ * to pause the crawler by calling {@apilink AutoscaledPool.pause|`autoscaledPool.pause()`}
+ * or to abort it by calling {@apilink AutoscaledPool.abort|`autoscaledPool.abort()`}.
*/
autoscaledPool?: AutoscaledPool;
/**
- * Default {@link Router} instance that will be used if we don't specify any {@link BasicCrawlerOptions.requestHandler|`requestHandler`}.
- * See {@link Router.addHandler|`router.addHandler()`} and {@link Router.addDefaultHandler|`router.addDefaultHandler()`}.
+ * Default {@apilink Router} instance that will be used if we don't specify any {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`}.
+ * See {@apilink Router.addHandler|`router.addHandler()`} and {@apilink Router.addDefaultHandler|`router.addDefaultHandler()`}.
*/
readonly router: RouterHandler<Context> = Router.create<Context>();
@@ -547,7 +547,7 @@ export class BasicCrawler<Context extends CrawlingContext = BasicCrawlingContext
/**
* Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
* We can use the `requests` parameter to enqueue the initial requests - it is a shortcut for
- * running {@link BasicCrawler.addRequests|`crawler.addRequests()`} before the {@link BasicCrawler.run|`crawler.run()`}.
+ * running {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} before the {@apilink BasicCrawler.run|`crawler.run()`}.
*
* @param [requests] The requests to add
* @param [options] Options for the request queue
@@ -1107,8 +1107,8 @@ export interface CrawlerAddRequestsResult {
/**
* A promise which will resolve with the rest of the requests that were added to the queue.
*
- * Alternatively, we can set {@link CrawlerAddRequestsOptions.waitForAllRequestsToBeAdded|`waitForAllRequestsToBeAdded`} to `true`
- * in the {@link BasicCrawler.addRequests|`crawler.addRequests()`} options.
+ * Alternatively, we can set {@apilink CrawlerAddRequestsOptions.waitForAllRequestsToBeAdded|`waitForAllRequestsToBeAdded`} to `true`
+ * in the {@apilink BasicCrawler.addRequests|`crawler.addRequests()`} options.
*
* **Example:**
*
@@ -1133,9 +1133,9 @@ interface HandlePropertyNameChangeData<New, Old> {
}
/**
- * Creates new {@link Router} instance that works based on request labels.
- * This instance can then serve as a {@link BasicCrawlerOptions.requestHandler|`requestHandler`} of our {@link BasicCrawler}.
- * Defaults to the {@link BasicCrawlingContext}.
+ * Creates new {@apilink Router} instance that works based on request labels.
+ * This instance can then serve as a {@apilink BasicCrawlerOptions.requestHandler|`requestHandler`} of our {@apilink BasicCrawler}.
+ * Defaults to the {@apilink BasicCrawlingContext}.
*
* > Serves as a shortcut for using `Router.create<BasicCrawlingContext>()`.
*
diff --git a/packages/basic-crawler/src/internals/constants.ts b/packages/basic-crawler/src/internals/constants.ts
index 2dea9f508..56c4aa46e 100644
--- a/packages/basic-crawler/src/internals/constants.ts
+++ b/packages/basic-crawler/src/internals/constants.ts
@@ -1,6 +1,6 @@
/**
- * Additional number of seconds used in {@link CheerioCrawler} and {@link BrowserCrawler} to set a reasonable
- * {@link BasicCrawlerOptions.requestHandlerTimeoutSecs|`requestHandlerTimeoutSecs`} for {@link BasicCrawler}
+ * Additional number of seconds used in {@apilink CheerioCrawler} and {@apilink BrowserCrawler} to set a reasonable
+ * {@apilink BasicCrawlerOptions.requestHandlerTimeoutSecs|`requestHandlerTimeoutSecs`} for {@apilink BasicCrawler}
* that would not impare functionality (not timeout before crawlers).
*/
export const BASIC_CRAWLER_TIMEOUT_BUFFER_SECS = 10;
diff --git a/packages/browser-crawler/README.md b/packages/browser-crawler/README.md
new file mode 100644
index 000000000..e9d635d3f
--- /dev/null
+++ b/packages/browser-crawler/README.md
@@ -0,0 +1,18 @@
+# `@crawlee/browser`
+
+Provides a simple framework for parallel crawling of web pages using headless browsers with [Puppeteer](https://github.com/puppeteer/puppeteer) and [Playwright](https://github.com/microsoft/playwright). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+Since `BrowserCrawler` uses headless (or even headful) browsers to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, we should consider using the [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.
+
+The source URLs are represented by the [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from the [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/api/browser-crawler/class/BrowserCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/api/browser-crawler/class/BrowserCrawler#run) function is provided.
+
+If both [`requestList`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+`BrowserCrawler` opens a new browser page (i.e. tab or window) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl and then calls the function provided by user as the [`requestHandler`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestHandler) option.
+
+New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class.
+All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#autoscaledPoolOptions) parameter of the `BrowserCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) constructor are available directly in the `BrowserCrawler` constructor.
+
+> *NOTE:* the pool of browser instances is internally managed by the [BrowserPool](https://crawlee.dev/api/browser-pool/class/BrowserPool) class.
diff --git a/packages/browser-crawler/src/internals/browser-crawler.ts b/packages/browser-crawler/src/internals/browser-crawler.ts
index e4c9f2b68..687fb98f4 100644
--- a/packages/browser-crawler/src/internals/browser-crawler.ts
+++ b/packages/browser-crawler/src/internals/browser-crawler.ts
@@ -80,15 +80,15 @@ export interface BrowserCrawlerOptions<
/**
* Function that is called to process each request.
*
- * The function receives the {@link BrowserCrawlingContext}
+ * The function receives the {@apilink BrowserCrawlingContext}
* (actual context will be enhanced with the crawler specific properties) as an argument, where:
- * - {@link BrowserCrawlingContext.request|`request`} is an instance of the {@link Request} object
+ * - {@apilink BrowserCrawlingContext.request|`request`} is an instance of the {@apilink Request} object
* with details about the URL to open, HTTP method etc;
- * - {@link BrowserCrawlingContext.page|`page`} is an instance of the
+ * - {@apilink BrowserCrawlingContext.page|`page`} is an instance of the
* Puppeteer [Page](https://pptr.dev/api/puppeteer.page) or
* Playwright [Page](https://playwright.dev/docs/api/class-page);
- * - {@link BrowserCrawlingContext.browserController|`browserController`} is an instance of the {@link BrowserController};
- * - {@link BrowserCrawlingContext.response|`response`} is an instance of the
+ * - {@apilink BrowserCrawlingContext.browserController|`browserController`} is an instance of the {@apilink BrowserController};
+ * - {@apilink BrowserCrawlingContext.response|`response`} is an instance of the
* Puppeteer [Response](https://pptr.dev/api/puppeteer.httpresponse) or
* Playwright [Response](https://playwright.dev/docs/api/class-response),
* which is the main resource response as returned by the respective `page.goto()` function.
@@ -96,28 +96,28 @@ export interface BrowserCrawlerOptions<
* The function must return a promise, which is then awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to the {@link BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * request later, up to the {@apilink BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
* If all the retries fail, the crawler calls the function
- * provided to the {@link BrowserCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
+ * provided to the {@apilink BrowserCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
* To make this work, we should **always**
* let our function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
+ * {@apilink Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
*/
requestHandler?: BrowserRequestHandler<Context>;
/**
* Function that is called to process each request.
*
- * The function receives the {@link BrowserCrawlingContext}
+ * The function receives the {@apilink BrowserCrawlingContext}
* (actual context will be enhanced with the crawler specific properties) as an argument, where:
- * - {@link BrowserCrawlingContext.request|`request`} is an instance of the {@link Request} object
+ * - {@apilink BrowserCrawlingContext.request|`request`} is an instance of the {@apilink Request} object
* with details about the URL to open, HTTP method etc;
- * - {@link BrowserCrawlingContext.page|`page`} is an instance of the
+ * - {@apilink BrowserCrawlingContext.page|`page`} is an instance of the
* Puppeteer [Page](https://pptr.dev/api/puppeteer.page) or
* Playwright [Page](https://playwright.dev/docs/api/class-page);
- * - {@link BrowserCrawlingContext.browserController|`browserController`} is an instance of the {@link BrowserController};
- * - {@link BrowserCrawlingContext.response|`response`} is an instance of the
+ * - {@apilink BrowserCrawlingContext.browserController|`browserController`} is an instance of the {@apilink BrowserController};
+ * - {@apilink BrowserCrawlingContext.response|`response`} is an instance of the
* Puppeteer [Response](https://pptr.dev/api/puppeteer.httpresponse) or
* Playwright [Response](https://playwright.dev/docs/api/class-response),
* which is the main resource response as returned by the respective `page.goto()` function.
@@ -125,13 +125,13 @@ export interface BrowserCrawlerOptions<
* The function must return a promise, which is then awaited by the crawler.
*
* If the function throws an exception, the crawler will try to re-crawl the
- * request later, up to the {@link BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * request later, up to the {@apilink BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
* If all the retries fail, the crawler calls the function
- * provided to the {@link BrowserCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
+ * provided to the {@apilink BrowserCrawlerOptions.failedRequestHandler|`failedRequestHandler`} parameter.
* To make this work, we should **always**
* let our function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
+ * {@apilink Request.pushErrorMessage|`Request.pushErrorMessage()`} function.
*
* @deprecated `handlePageFunction` has been renamed to `requestHandler` and will be removed in a future version.
* @ignore
@@ -140,11 +140,11 @@ export interface BrowserCrawlerOptions<
/**
* User-provided function that allows modifying the request object before it gets retried by the crawler.
- * It's executed before each retry for the requests that failed less than {@link BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
+ * It's executed before each retry for the requests that failed less than {@apilink BrowserCrawlerOptions.maxRequestRetries|`maxRequestRetries`} times.
*
- * The function receives the {@link BrowserCrawlingContext}
+ * The function receives the {@apilink BrowserCrawlingContext}
* (actual context will be enhanced with the crawler specific properties) as the first argument,
- * where the {@link BrowserCrawlingContext.request|`request`} corresponds to the request to be retried.
+ * where the {@apilink BrowserCrawlingContext.request|`request`} corresponds to the request to be retried.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*/
@@ -153,9 +153,9 @@ export interface BrowserCrawlerOptions<
/**
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
- * The function receives the {@link BrowserCrawlingContext}
+ * The function receives the {@apilink BrowserCrawlingContext}
* (actual context will be enhanced with the crawler specific properties) as the first argument,
- * where the {@link BrowserCrawlingContext.request|`request`} corresponds to the failed request.
+ * where the {@apilink BrowserCrawlingContext.request|`request`} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*/
@@ -164,9 +164,9 @@ export interface BrowserCrawlerOptions<
/**
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
- * The function receives the {@link BrowserCrawlingContext}
+ * The function receives the {@apilink BrowserCrawlingContext}
* (actual context will be enhanced with the crawler specific properties) as the first argument,
- * where the {@link BrowserCrawlingContext.request|`request`} corresponds to the failed request.
+ * where the {@apilink BrowserCrawlingContext.request|`request`} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*
@@ -176,7 +176,7 @@ export interface BrowserCrawlerOptions<
handleFailedRequestFunction?: BrowserErrorHandler<Context>;
/**
- * Custom options passed to the underlying {@link BrowserPool} constructor.
+ * Custom options passed to the underlying {@apilink BrowserPool} constructor.
* We can tweak those to fine-tune browser management.
*/
browserPoolOptions?: Partial<BrowserPoolOptions> & Partial<BrowserPoolHooks<__BrowserControllerReturn, __LaunchContextReturn>>;
@@ -239,7 +239,7 @@ export interface BrowserCrawlerOptions<
/**
* Whether to run browser in headless mode. Defaults to `true`.
- * Can be also set via {@link Configuration}.
+ * Can be also set via {@apilink Configuration}.
*/
headless?: boolean;
}
@@ -253,33 +253,33 @@ export interface BrowserCrawlerOptions<
*
* Since `BrowserCrawler` uses headless (or even headful) browsers to download web pages and extract data,
* it is useful for crawling of websites that require to execute JavaScript.
- * If the target website doesn't need JavaScript, we should consider using the {@link CheerioCrawler},
+ * If the target website doesn't need JavaScript, we should consider using the {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented by the {@link Request} objects that are fed from the {@link RequestList} or {@link RequestQueue} instances
- * provided by the {@link BrowserCrawlerOptions.requestList|`requestList`} or {@link BrowserCrawlerOptions.requestQueue|`requestQueue`}
+ * The source URLs are represented by the {@apilink Request} objects that are fed from the {@apilink RequestList} or {@apilink RequestQueue} instances
+ * provided by the {@apilink BrowserCrawlerOptions.requestList|`requestList`} or {@apilink BrowserCrawlerOptions.requestQueue|`requestQueue`}
* constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided,
- * the crawler will open the default request queue either when the {@link BrowserCrawler.addRequests|`crawler.addRequests()`} function is called,
- * or if `requests` parameter (representing the initial requests) of the {@link BrowserCrawler.run|`crawler.run()`} function is provided.
+ * the crawler will open the default request queue either when the {@apilink BrowserCrawler.addRequests|`crawler.addRequests()`} function is called,
+ * or if `requests` parameter (representing the initial requests) of the {@apilink BrowserCrawler.run|`crawler.run()`} function is provided.
*
- * If both {@link BrowserCrawlerOptions.requestList|`requestList`} and {@link BrowserCrawlerOptions.requestQueue|`requestQueue`} options are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to the {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * If both {@apilink BrowserCrawlerOptions.requestList|`requestList`} and {@apilink BrowserCrawlerOptions.requestQueue|`requestQueue`} options are used,
+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
+ * to the {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
- * The crawler finishes when there are no more {@link Request} objects to crawl.
+ * The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
- * `BrowserCrawler` opens a new browser page (i.e. tab or window) for each {@link Request} object to crawl
- * and then calls the function provided by user as the {@link BrowserCrawlerOptions.requestHandler|`requestHandler`} option.
+ * `BrowserCrawler` opens a new browser page (i.e. tab or window) for each {@apilink Request} object to crawl
+ * and then calls the function provided by user as the {@apilink BrowserCrawlerOptions.requestHandler|`requestHandler`} option.
*
* New pages are only opened when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the {@link BrowserCrawlerOptions.autoscaledPoolOptions|`autoscaledPoolOptions`}
+ * using the functionality provided by the {@apilink AutoscaledPool} class.
+ * All {@apilink AutoscaledPool} configuration options can be passed to the {@apilink BrowserCrawlerOptions.autoscaledPoolOptions|`autoscaledPoolOptions`}
* parameter of the `BrowserCrawler` constructor.
- * For user convenience, the {@link AutoscaledPoolOptions.minConcurrency|`minConcurrency`} and
- * {@link AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} options of the
- * underlying {@link AutoscaledPool} constructor are available directly in the `BrowserCrawler` constructor.
+ * For user convenience, the {@apilink AutoscaledPoolOptions.minConcurrency|`minConcurrency`} and
+ * {@apilink AutoscaledPoolOptions.maxConcurrency|`maxConcurrency`} options of the
+ * underlying {@apilink AutoscaledPool} constructor are available directly in the `BrowserCrawler` constructor.
*
- * > *NOTE:* the pool of browser instances is internally managed by the {@link BrowserPool} class.
+ * > *NOTE:* the pool of browser instances is internally managed by the {@apilink BrowserPool} class.
*
* @category Crawlers
*/
@@ -290,13 +290,13 @@ export abstract class BrowserCrawler<
GoToOptions extends Record<PropertyKey, any> = Dictionary
> extends BasicCrawler<Context> {
/**
- * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * A reference to the underlying {@apilink ProxyConfiguration} class that manages the crawler's proxies.
* Only available if used by the crawler.
*/
proxyConfiguration?: ProxyConfiguration;
/**
- * A reference to the underlying {@link BrowserPool} class that manages the crawler's browsers.
+ * A reference to the underlying {@apilink BrowserPool} class that manages the crawler's browsers.
*/
browserPool: BrowserPool<InternalBrowserPoolOptions>;
diff --git a/packages/browser-crawler/src/internals/browser-launcher.ts b/packages/browser-crawler/src/internals/browser-launcher.ts
index 4025ba6e2..f14ebb4eb 100644
--- a/packages/browser-crawler/src/internals/browser-launcher.ts
+++ b/packages/browser-crawler/src/internals/browser-launcher.ts
@@ -21,7 +21,7 @@ export interface BrowserLaunchContext<TOptions, Launcher> extends BrowserPluginO
proxyUrl?: string;
/**
- * If `true` and the `executablePath` option of {@link BrowserLaunchContext.launchOptions|`launchOptions`} is not set,
+ * If `true` and the `executablePath` option of {@apilink BrowserLaunchContext.launchOptions|`launchOptions`} is not set,
* the launcher will launch full Google Chrome browser available on the machine
* rather than the bundled Chromium. The path to Chrome executable
* is taken from the `CRAWLEE_CHROME_EXECUTABLE_PATH` environment variable if provided,
diff --git a/packages/browser-pool/src/abstract-classes/browser-plugin.ts b/packages/browser-pool/src/abstract-classes/browser-plugin.ts
index 01c7e48bb..a6588c00f 100644
--- a/packages/browser-pool/src/abstract-classes/browser-plugin.ts
+++ b/packages/browser-pool/src/abstract-classes/browser-plugin.ts
@@ -44,7 +44,7 @@ export interface BrowserPluginOptions<LibraryOptions> {
* Options that will be passed down to the automation library. E.g.
* `puppeteer.launch(launchOptions);`. This is a good place to set
* options that you want to apply as defaults. To dynamically override
- * those options per-browser, see the `preLaunchHooks` of {@link BrowserPool}.
+ * those options per-browser, see the `preLaunchHooks` of {@apilink BrowserPool}.
*/
launchOptions?: LibraryOptions;
/**
@@ -83,7 +83,7 @@ export interface CreateLaunchContextOptions<
* The `BrowserPlugin` serves two purposes. First, it is the base class that
* specialized controllers like `PuppeteerPlugin` or `PlaywrightPlugin` extend.
* Second, it allows the user to configure the automation libraries and
- * feed them to {@link BrowserPool} for use.
+ * feed them to {@apilink BrowserPool} for use.
*/
export abstract class BrowserPlugin<
Library extends CommonLibrary = CommonLibrary,
diff --git a/packages/browser-pool/src/browser-pool.ts b/packages/browser-pool/src/browser-pool.ts
index 0d600580c..b2e714a20 100644
--- a/packages/browser-pool/src/browser-pool.ts
+++ b/packages/browser-pool/src/browser-pool.ts
@@ -87,16 +87,16 @@ export interface BrowserPoolOptions<Plugin extends BrowserPlugin = BrowserPlugin
* Pre-launch hooks are executed just before a browser is launched and provide
* a good opportunity to dynamically change the launch options.
* The hooks are called with two arguments:
- * `pageId`: `string` and `launchContext`: {@link LaunchContext}
+ * `pageId`: `string` and `launchContext`: {@apilink LaunchContext}
*/
export type PreLaunchHook<LC extends LaunchContext> = (pageId: string, launchContext: LC) => void | Promise<void>;
/**
* Post-launch hooks are executed as soon as a browser is launched.
* The hooks are called with two arguments:
- * `pageId`: `string` and `browserController`: {@link BrowserController}
+ * `pageId`: `string` and `browserController`: {@apilink BrowserController}
* To guarantee order of execution before other hooks in the same browser,
- * the {@link BrowserController} methods cannot be used until the post-launch
+ * the {@apilink BrowserController} methods cannot be used until the post-launch
* hooks complete. If you attempt to call `await browserController.close()` from
* a post-launch hook, it will deadlock the process. This API is subject to change.
*/
@@ -106,7 +106,7 @@ export type PostLaunchHook<BC extends BrowserController> = (pageId: string, brow
* Pre-page-create hooks are executed just before a new page is created. They
* are useful to make dynamic changes to the browser before opening a page.
* The hooks are called with three arguments:
- * `pageId`: `string`, `browserController`: {@link BrowserController} and
+ * `pageId`: `string`, `browserController`: {@apilink BrowserController} and
* `pageOptions`: `object|undefined` - This only works if the underlying `BrowserController` supports new page options.
* So far, new page options are only supported by `PlaywrightController`.
* If the page options are not supported by `BrowserController` the `pageOptions` argument is `undefined`.
@@ -122,7 +122,7 @@ export type PrePageCreateHook<
* place to make changes to a page that you would like to apply to all
* pages. Such as injecting a JavaScript library into all pages.
* The hooks are called with two arguments:
- * `page`: `Page` and `browserController`: {@link BrowserController}
+ * `page`: `Page` and `browserController`: {@apilink BrowserController}
*/
export type PostPageCreateHook<
BC extends BrowserController,
@@ -134,7 +134,7 @@ export type PostPageCreateHook<
* in a page that's about to be closed, such as saving a snapshot or updating
* state.
* The hooks are called with two arguments:
- * `page`: `Page` and `browserController`: {@link BrowserController}
+ * `page`: `Page` and `browserController`: {@apilink BrowserController}
*/
export type PrePageCloseHook<
BC extends BrowserController,
@@ -144,7 +144,7 @@ export type PrePageCloseHook<
/**
* Post-page-close hooks allow you to do page related clean up.
* The hooks are called with two arguments:
- * `pageId`: `string` and `browserController`: {@link BrowserController}
+ * `pageId`: `string` and `browserController`: {@apilink BrowserController}
*/
export type PostPageCloseHook<BC extends BrowserController> = (pageId: string, browserController: BC) => void | Promise<void>;
@@ -157,15 +157,15 @@ export interface BrowserPoolHooks<
* Pre-launch hooks are executed just before a browser is launched and provide
* a good opportunity to dynamically change the launch options.
* The hooks are called with two arguments:
- * `pageId`: `string` and `launchContext`: {@link LaunchContext}
+ * `pageId`: `string` and `launchContext`: {@apilink LaunchContext}
*/
preLaunchHooks?: PreLaunchHook<LC>[];
/**
* Post-launch hooks are executed as soon as a browser is launched.
* The hooks are called with two arguments:
- * `pageId`: `string` and `browserController`: {@link BrowserController}
+ * `pageId`: `string` and `browserController`: {@apilink BrowserController}
* To guarantee order of execution before other hooks in the same browser,
- * the {@link BrowserController} methods cannot be used until the post-launch
+ * the {@apilink BrowserController} methods cannot be used until the post-launch
* hooks complete. If you attempt to call `await browserController.close()` from
* a post-launch hook, it will deadlock the process. This API is subject to change.
*/
@@ -174,7 +174,7 @@ export interface BrowserPoolHooks<
* Pre-page-create hooks are executed just before a new page is created. They
* are useful to make dynamic changes to the browser before opening a page.
* The hooks are called with three arguments:
- * `pageId`: `string`, `browserController`: {@link BrowserController} and
+ * `pageId`: `string`, `browserController`: {@apilink BrowserController} and
* `pageOptions`: `object|undefined` - This only works if the underlying `BrowserController` supports new page options.
* So far, new page options are only supported by `PlaywrightController`.
* If the page options are not supported by `BrowserController` the `pageOptions` argument is `undefined`.
@@ -186,7 +186,7 @@ export interface BrowserPoolHooks<
* place to make changes to a page that you would like to apply to all
* pages. Such as injecting a JavaScript library into all pages.
* The hooks are called with two arguments:
- * `page`: `Page` and `browserController`: {@link BrowserController}
+ * `page`: `Page` and `browserController`: {@apilink BrowserController}
*/
postPageCreateHooks?: PostPageCreateHook<BC, PR>[];
/**
@@ -194,13 +194,13 @@ export interface BrowserPoolHooks<
* in a page that's about to be closed, such as saving a snapshot or updating
* state.
* The hooks are called with two arguments:
- * `page`: `Page` and `browserController`: {@link BrowserController}
+ * `page`: `Page` and `browserController`: {@apilink BrowserController}
*/
prePageCloseHooks?: PrePageCloseHook<BC, PR>[];
/**
* Post-page-close hooks allow you to do page related clean up.
* The hooks are called with two arguments:
- * `pageId`: `string` and `browserController`: {@link BrowserController}
+ * `pageId`: `string` and `browserController`: {@apilink BrowserController}
*/
postPageCloseHooks?: PostPageCloseHook<BC>[];
}
@@ -395,7 +395,7 @@ export class BrowserPool<
}
/**
- * Unlike {@link newPage}, `newPageInNewBrowser` always launches a new
+ * Unlike {@apilink newPage}, `newPageInNewBrowser` always launches a new
* browser to open the page in. Use the `launchOptions` option to
* configure the new browser.
*/
@@ -450,7 +450,7 @@ export class BrowserPool<
}
/**
- * Retrieves a {@link BrowserController} for a given page. This is useful
+ * Retrieves a {@apilink BrowserController} for a given page. This is useful
* when you're working only with pages and need to access the browser
* manipulation functionality.
*
diff --git a/packages/cheerio-crawler/README.md b/packages/cheerio-crawler/README.md
new file mode 100644
index 000000000..687ed90f5
--- /dev/null
+++ b/packages/cheerio-crawler/README.md
@@ -0,0 +1,55 @@
+# `@crawlee/cheerio`
+
+Provides a framework for the parallel crawling of web pages using plain HTTP requests and [cheerio](https://www.npmjs.com/package/cheerio) HTML parser. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+Since `CheerioCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.
+
+`CheerioCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio) and then invokes the user-provided [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler) to extract page data using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.
+
+The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [CheerioCrawlerOptions.requestList](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) or [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) constructor options, respectively.
+
+If both [CheerioCrawlerOptions.requestList](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) and [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+We can use the `preNavigationHooks` to adjust `gotOptions`:
+
+```
+preNavigationHooks: [
+ (crawlingContext, gotOptions) => {
+ // ...
+ },
+]
+```
+
+By default, `CheerioCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [CheerioCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For details, see [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler).
+
+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `CheerioCrawler` constructor.
+
+## Example usage
+
+```ts
+const crawler = new CheerioCrawler({
+ requestList,
+ async requestHandler({ request, response, body, contentType, $ }) {
+ const data = [];
+
+ // Do some data extraction from the page with Cheerio.
+ $('.some-collection').each((index, el) => {
+ data.push({ title: $(el).find('.some-title').text() });
+ });
+
+ // Save the data to dataset.
+ await Dataset.pushData({
+ url: request.url,
+ html: body,
+ data,
+ })
+ },
+});
+
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+]);
+```
diff --git a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
index 71257d4f7..1a8feae6e 100644
--- a/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
+++ b/packages/cheerio-crawler/src/internals/cheerio-crawler.ts
@@ -59,16 +59,16 @@ export interface CheerioCrawlerOptions<
* User-provided function that performs the logic of the crawler. It is called for each page
* loaded and parsed by the crawler.
*
- * The function receives the {@link CheerioCrawlingContext} as an argument,
- * where the {@link CheerioCrawlingContext.request} instance represents the URL to crawl.
+ * The function receives the {@apilink CheerioCrawlingContext} as an argument,
+ * where the {@apilink CheerioCrawlingContext.request} instance represents the URL to crawl.
*
- * Type of {@link CheerioCrawlingContext.body} depends on the `Content-Type` header of the web page:
+ * Type of {@apilink CheerioCrawlingContext.body} depends on the `Content-Type` header of the web page:
* - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types
* - Buffer for others MIME content types
*
* Parsed `Content-Type` header using
* [content-type package](https://www.npmjs.com/package/content-type)
- * is stored in {@link CheerioCrawlingContext.contentType}`.
+ * is stored in {@apilink CheerioCrawlingContext.contentType}`.
*
* Cheerio is available only for HTML and XML content types.
*
@@ -81,7 +81,7 @@ export interface CheerioCrawlerOptions<
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage} function.
+ * {@apilink Request.pushErrorMessage} function.
*/
requestHandler?: CheerioRequestHandler<UserData, JSONData>;
@@ -89,16 +89,16 @@ export interface CheerioCrawlerOptions<
* User-provided function that performs the logic of the crawler. It is called for each page
* loaded and parsed by the crawler.
*
- * The function receives the {@link CheerioCrawlingContext} as an argument,
- * where the {@link CheerioCrawlingContext.request} instance represents the URL to crawl.
+ * The function receives the {@apilink CheerioCrawlingContext} as an argument,
+ * where the {@apilink CheerioCrawlingContext.request} instance represents the URL to crawl.
*
- * Type of {@link CheerioCrawlingContext.body} depends on the `Content-Type` header of the web page:
+ * Type of {@apilink CheerioCrawlingContext.body} depends on the `Content-Type` header of the web page:
* - String for `text/html`, `application/xhtml+xml`, `application/xml` MIME content types
* - Buffer for others MIME content types
*
* Parsed `Content-Type` header using
* [content-type package](https://www.npmjs.com/package/content-type)
- * is stored in {@link CheerioCrawlingContext.contentType}`.
+ * is stored in {@apilink CheerioCrawlingContext.contentType}`.
*
* Cheerio is available only for HTML and XML content types.
*
@@ -111,7 +111,7 @@ export interface CheerioCrawlerOptions<
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage} function.
+ * {@apilink Request.pushErrorMessage} function.
*
* @deprecated `handlePageFunction` has been renamed to `requestHandler` and will be removed in a future version.
* @ignore
@@ -139,8 +139,8 @@ export interface CheerioCrawlerOptions<
* User-provided function that allows modifying the request object before it gets retried by the crawler.
* It's executed before each retry for the requests that failed less than `option.maxRequestRetries` times.
*
- * The function receives the {@link CheerioCrawlingContext} as the first argument,
- * where the {@link CheerioCrawlingContext.request} corresponds to the request to be retried.
+ * The function receives the {@apilink CheerioCrawlingContext} as the first argument,
+ * where the {@apilink CheerioCrawlingContext.request} corresponds to the request to be retried.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*/
@@ -149,8 +149,8 @@ export interface CheerioCrawlerOptions<
/**
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
- * The function receives the {@link CheerioCrawlingContext} as the first argument,
- * where the {@link CheerioCrawlingContext.request} corresponds to the failed request.
+ * The function receives the {@apilink CheerioCrawlingContext} as the first argument,
+ * where the {@apilink CheerioCrawlingContext.request} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*
@@ -162,8 +162,8 @@ export interface CheerioCrawlerOptions<
/**
* A function to handle requests that failed more than `option.maxRequestRetries` times.
*
- * The function receives the {@link CheerioCrawlingContext} as the first argument,
- * where the {@link CheerioCrawlingContext.request} corresponds to the failed request.
+ * The function receives the {@apilink CheerioCrawlingContext} as the first argument,
+ * where the {@apilink CheerioCrawlingContext.request} corresponds to the failed request.
* Second argument is the `Error` instance that
* represents the last error thrown during processing of the request.
*
@@ -215,7 +215,7 @@ export interface CheerioCrawlerOptions<
* Sadly, there are some websites which use invalid headers. Those are encoded using the UTF-8 encoding.
* If those sites actually use a different encoding, the response will be corrupted. You can use
* `suggestResponseEncoding` to fall back to a certain encoding, if you know that your target website uses it.
- * To force a certain encoding, disregarding the response headers, use {@link CheerioCrawlerOptions.forceResponseEncoding}
+ * To force a certain encoding, disregarding the response headers, use {@apilink CheerioCrawlerOptions.forceResponseEncoding}
* ```
* // Will fall back to windows-1250 encoding if none found
* suggestResponseEncoding: 'windows-1250'
@@ -226,7 +226,7 @@ export interface CheerioCrawlerOptions<
/**
* By default `CheerioCrawler` will extract correct encoding from the HTTP response headers. Use `forceResponseEncoding`
* to force a certain encoding, disregarding the response headers.
- * To only provide a default for missing encodings, use {@link CheerioCrawlerOptions.suggestResponseEncoding}
+ * To only provide a default for missing encodings, use {@apilink CheerioCrawlerOptions.suggestResponseEncoding}
* ```
* // Will force windows-1250 encoding even if headers say otherwise
* forceResponseEncoding: 'windows-1250'
@@ -294,23 +294,23 @@ export interface CheerioCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOpti
*
* Since `CheerioCrawler` uses raw HTTP requests to download web pages,
* it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript
- * to display the content, you might need to use {@link PuppeteerCrawler} or {@link PlaywrightCrawler} instead,
+ * to display the content, you might need to use {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler} instead,
* because it loads the pages using full-featured headless Chrome browser.
*
* `CheerioCrawler` downloads each URL using a plain HTTP request,
* parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio)
- * and then invokes the user-provided {@link CheerioCrawlerOptions.requestHandler} to extract page data
+ * and then invokes the user-provided {@apilink CheerioCrawlerOptions.requestHandler} to extract page data
* using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.
*
- * The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the {@link CheerioCrawlerOptions.requestList}
- * or {@link CheerioCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from
+ * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink CheerioCrawlerOptions.requestList}
+ * or {@apilink CheerioCrawlerOptions.requestQueue} constructor options, respectively.
*
- * If both {@link CheerioCrawlerOptions.requestList} and {@link CheerioCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * If both {@apilink CheerioCrawlerOptions.requestList} and {@apilink CheerioCrawlerOptions.requestQueue} are used,
+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
+ * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
- * The crawler finishes when there are no more {@link Request} objects to crawl.
+ * The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
* We can use the `preNavigationHooks` to adjust `gotOptions`:
*
@@ -325,28 +325,20 @@ export interface CheerioCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOpti
* By default, `CheerioCrawler` only processes web pages with the `text/html`
* and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header),
* and skips pages with other content types. If you want the crawler to process other content types,
- * use the {@link CheerioCrawlerOptions.additionalMimeTypes} constructor option.
+ * use the {@apilink CheerioCrawlerOptions.additionalMimeTypes} constructor option.
* Beware that the parsing behavior differs for HTML, XML, JSON and other types of content.
- * For details, see {@link CheerioCrawlerOptions.requestHandler}.
+ * For details, see {@apilink CheerioCrawlerOptions.requestHandler}.
*
* New requests are only dispatched when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * using the functionality provided by the {@apilink AutoscaledPool} class.
+ * All {@apilink AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
* parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
- * {@link AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.
+ * {@apilink AutoscaledPool} options are available directly in the `CheerioCrawler` constructor.
*
* **Example usage:**
*
* ```javascript
- * // Prepare a list of URLs to crawl
- * const requestList = await RequestList.open(null, [
- * { url: 'http://www.example.com/page-1' },
- * { url: 'http://www.example.com/page-2' },
- * ]);
- *
- * // Crawl the URLs
* const crawler = new CheerioCrawler({
- * requestList,
* async requestHandler({ request, response, body, contentType, $ }) {
* const data = [];
*
@@ -364,13 +356,16 @@ export interface CheerioCrawlerEnqueueLinksOptions extends Omit<EnqueueLinksOpti
* },
* });
*
- * await crawler.run();
+ * await crawler.run([
+ * 'http://www.example.com/page-1',
+ * 'http://www.example.com/page-2',
+ * ]);
* ```
* @category Crawlers
*/
export class CheerioCrawler extends BasicCrawler<CheerioCrawlingContext> {
/**
- * A reference to the underlying {@link ProxyConfiguration} class that manages the crawler's proxies.
+ * A reference to the underlying {@apilink ProxyConfiguration} class that manages the crawler's proxies.
* Only available if used by the crawler.
*/
proxyConfiguration?: ProxyConfiguration;
@@ -979,9 +974,9 @@ function addResponsePropertiesToStream(stream: GotRequest) {
}
/**
- * Creates new {@link Router} instance that works based on request labels.
- * This instance can then serve as a `requestHandler` of your {@link CheerioCrawler}.
- * Defaults to the {@link CheerioCrawlingContext}.
+ * Creates new {@apilink Router} instance that works based on request labels.
+ * This instance can then serve as a `requestHandler` of your {@apilink CheerioCrawler}.
+ * Defaults to the {@apilink CheerioCrawlingContext}.
*
* > Serves as a shortcut for using `Router.create<CheerioCrawlingContext>()`.
*
diff --git a/packages/core/README.md b/packages/core/README.md
new file mode 100644
index 000000000..750ac27a1
--- /dev/null
+++ b/packages/core/README.md
@@ -0,0 +1,43 @@
+# `@crawlee/core`
+
+Core set of classes required for Crawlee.
+
+The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:
+
+- `@crawlee/core`: the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
+- `@crawlee/basic`: exports `BasicCrawler`
+- `@crawlee/cheerio`: exports `CheerioCrawler`
+- `@crawlee/browser`: exports `BrowserCrawler` (which is used for creating `@crawlee/playwright` and `@crawlee/puppeteer`)
+- `@crawlee/playwright`: exports `PlaywrightCrawler`
+- `@crawlee/puppeteer`: exports `PuppeteerCrawler`
+- `@crawlee/memory-storage`: `@apify/storage-local` alternative
+- `@crawlee/browser-pool`: previously `browser-pool` package
+- `@crawlee/utils`: utility methods
+- `@crawlee/types`: holds TS interfaces mainly about the `StorageClient`
+
+## Installing Crawlee
+
+Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. `@crawlee/playwright` if you plan on using `playwright` - it already contains everything from the `@crawlee/browser` package, which includes everything from `@crawlee/basic`, which includes everything from `@crawlee/core`.
+
+If we don't care much about additional code being pulled in, we can just use the `crawlee` meta-package, which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
+
+```bash
+npm install crawlee
+```
+
+Or if all we need is cheerio support, we can install only `@crawlee/cheerio`.
+
+```bash
+npm install @crawlee/cheerio
+```
+
+When using `playwright` or `puppeteer`, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.
+
+```bash
+npm install crawlee playwright
+# or npm install @crawlee/playwright playwright
+```
+
+Alternatively we can also use the `crawlee` meta-package which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
+
+> Sometimes you might want to use some utility methods from `@crawlee/utils`, so you might want to install that as well. This package contains some utilities that were previously available under `Apify.utils`. Browser related utilities can be also found in the crawler packages (e.g. `@crawlee/playwright`).
diff --git a/packages/core/src/autoscaling/autoscaled_pool.ts b/packages/core/src/autoscaling/autoscaled_pool.ts
index 0ffcc9675..6a9500678 100644
--- a/packages/core/src/autoscaling/autoscaled_pool.ts
+++ b/packages/core/src/autoscaling/autoscaled_pool.ts
@@ -30,7 +30,7 @@ export interface AutoscaledPoolOptions {
* If it resolves to `true` then the pool's run finishes. Being called only
* when there are no tasks being processed means that as long as `isTaskReadyFunction()`
* keeps resolving to `true`, `isFinishedFunction()` will never be called.
- * To abort a run, use the {@link AutoscaledPool.abort} method.
+ * To abort a run, use the {@apilink AutoscaledPool.abort} method.
*/
isFinishedFunction?: () => Promise<boolean>;
@@ -105,13 +105,13 @@ export interface AutoscaledPoolOptions {
taskTimeoutSecs?: number;
/**
- * Options to be passed down to the {@link Snapshotter} constructor. This is useful for fine-tuning
+ * Options to be passed down to the {@apilink Snapshotter} constructor. This is useful for fine-tuning
* the snapshot intervals and history.
*/
snapshotterOptions?: SnapshotterOptions;
/**
- * Options to be passed down to the {@link SystemStatus} constructor. This is useful for fine-tuning
+ * Options to be passed down to the {@apilink SystemStatus} constructor. This is useful for fine-tuning
* the system status reports. If a custom snapshotter is set in the options, it will be used
* by the pool.
*/
@@ -131,22 +131,22 @@ export interface AutoscaledPoolOptions {
* The pool only starts new tasks if there is enough free CPU and memory available
* and the Javascript event loop is not blocked.
*
- * The information about the CPU and memory usage is obtained by the {@link Snapshotter} class,
+ * The information about the CPU and memory usage is obtained by the {@apilink Snapshotter} class,
* which makes regular snapshots of system resources that may be either local
* or from the Apify cloud infrastructure in case the process is running on the Apify platform.
- * Meaningful data gathered from these snapshots is provided to `AutoscaledPool` by the {@link SystemStatus} class.
+ * Meaningful data gathered from these snapshots is provided to `AutoscaledPool` by the {@apilink SystemStatus} class.
*
* Before running the pool, you need to implement the following three functions:
- * {@link AutoscaledPoolOptions.runTaskFunction},
- * {@link AutoscaledPoolOptions.isTaskReadyFunction} and
- * {@link AutoscaledPoolOptions.isFinishedFunction}.
+ * {@apilink AutoscaledPoolOptions.runTaskFunction},
+ * {@apilink AutoscaledPoolOptions.isTaskReadyFunction} and
+ * {@apilink AutoscaledPoolOptions.isFinishedFunction}.
*
- * The auto-scaled pool is started by calling the {@link AutoscaledPool.run} function.
- * The pool periodically queries the {@link AutoscaledPoolOptions.isTaskReadyFunction} function
+ * The auto-scaled pool is started by calling the {@apilink AutoscaledPool.run} function.
+ * The pool periodically queries the {@apilink AutoscaledPoolOptions.isTaskReadyFunction} function
* for more tasks, managing optimal concurrency, until the function resolves to `false`. The pool then queries
- * the {@link AutoscaledPoolOptions.isFinishedFunction}. If it resolves to `true`, the run finishes after all running tasks complete.
+ * the {@apilink AutoscaledPoolOptions.isFinishedFunction}. If it resolves to `true`, the run finishes after all running tasks complete.
* If it resolves to `false`, it assumes there will be more tasks available later and keeps periodically querying for tasks.
- * If any of the tasks throws then the {@link AutoscaledPool.run} function rejects the promise with an error.
+ * If any of the tasks throws then the {@apilink AutoscaledPool.run} function rejects the promise with an error.
*
* The pool evaluates whether it should start a new task every time one of the tasks finishes
* and also in the interval set by the `options.maybeRunIntervalSecs` parameter.
@@ -385,7 +385,7 @@ export class AutoscaledPool {
/**
* Aborts the run of the auto-scaled pool and destroys it. The promise returned from
- * the {@link AutoscaledPool.run} function will immediately resolve, no more new tasks
+ * the {@apilink AutoscaledPool.run} function will immediately resolve, no more new tasks
* will be spawned and all running tasks will be left in their current state.
*
* Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion
@@ -404,13 +404,13 @@ export class AutoscaledPool {
/**
* Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish
- * (unlike abort, which terminates them). Used together with {@link AutoscaledPool.resume}
+ * (unlike abort, which terminates them). Used together with {@apilink AutoscaledPool.resume}
*
* The function's promise will resolve once all running tasks have completed and the pool
* is effectively idle. If the `timeoutSecs` argument is provided, the promise will reject
* with a timeout error after the `timeoutSecs` seconds.
*
- * The promise returned from the {@link AutoscaledPool.run} function will not resolve
+ * The promise returned from the {@apilink AutoscaledPool.run} function will not resolve
* when `.pause()` is invoked (unlike abort, which resolves it).
*/
async pause(timeoutSecs?: number): Promise<void> {
@@ -439,7 +439,7 @@ export class AutoscaledPool {
/**
* Resumes the operation of the autoscaled-pool by allowing more tasks to be run.
- * Used together with {@link AutoscaledPool.pause}
+ * Used together with {@apilink AutoscaledPool.pause}
*
* Tasks will automatically start running again in `options.maybeRunIntervalSecs`.
*/
diff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/core/src/autoscaling/snapshotter.ts
index 6ce5a16dd..8e0648096 100644
--- a/packages/core/src/autoscaling/snapshotter.ts
+++ b/packages/core/src/autoscaling/snapshotter.ts
@@ -75,7 +75,7 @@ interface ClientSnapshot { createdAt: Date; isOverloaded: boolean; rateLimitErro
* Creates snapshots of system resources at given intervals and marks the resource
* as either overloaded or not during the last interval. Keeps a history of the snapshots.
* It tracks the following resources: Memory, EventLoop, API and CPU.
- * The class is used by the {@link AutoscaledPool} class.
+ * The class is used by the {@apilink AutoscaledPool} class.
*
* When running on the Apify platform, the CPU and memory statistics are provided by the platform,
* as collected from the running Docker container. When running locally, `Snapshotter`
diff --git a/packages/core/src/autoscaling/system_status.ts b/packages/core/src/autoscaling/system_status.ts
index 2602103c4..06ce3bbbe 100644
--- a/packages/core/src/autoscaling/system_status.ts
+++ b/packages/core/src/autoscaling/system_status.ts
@@ -33,7 +33,7 @@ export interface SystemInfo {
export interface SystemStatusOptions {
/**
- * Defines max age of snapshots used in the {@link SystemStatus.getCurrentStatus} measurement.
+ * Defines max age of snapshots used in the {@apilink SystemStatus.getCurrentStatus} measurement.
* @default 5
*/
currentHistorySecs?: number;
@@ -95,25 +95,25 @@ export interface FinalStatistics {
}
/**
- * Provides a simple interface to reading system status from a {@link Snapshotter} instance.
- * It only exposes two functions {@link SystemStatus.getCurrentStatus}
- * and {@link SystemStatus.getHistoricalStatus}.
+ * Provides a simple interface to reading system status from a {@apilink Snapshotter} instance.
+ * It only exposes two functions {@apilink SystemStatus.getCurrentStatus}
+ * and {@apilink SystemStatus.getHistoricalStatus}.
* The system status is calculated using a weighted average of overloaded
* messages in the snapshots, with the weights being the time intervals
* between the snapshots. Each resource is calculated separately
* and the system is overloaded whenever at least one resource is overloaded.
- * The class is used by the {@link AutoscaledPool} class.
+ * The class is used by the {@apilink AutoscaledPool} class.
*
- * {@link SystemStatus.getCurrentStatus}
+ * {@apilink SystemStatus.getCurrentStatus}
* returns a boolean that represents the current status of the system.
* The length of the current timeframe in seconds is configurable
* by the `currentHistorySecs` option and represents the max age
* of snapshots to be considered for the calculation.
*
- * {@link SystemStatus.getHistoricalStatus}
+ * {@apilink SystemStatus.getHistoricalStatus}
* returns a boolean that represents the long-term status
* of the system. It considers the full snapshot history available
- * in the {@link Snapshotter} instance.
+ * in the {@apilink Snapshotter} instance.
* @category Scaling
*/
export class SystemStatus {
@@ -154,7 +154,7 @@ export class SystemStatus {
}
/**
- * Returns an {@link SystemInfo} object with the following structure:
+ * Returns an {@apilink SystemInfo} object with the following structure:
*
* ```javascript
* {
@@ -174,7 +174,7 @@ export class SystemStatus {
}
/**
- * Returns an {@link SystemInfo} object with the following structure:
+ * Returns an {@apilink SystemInfo} object with the following structure:
*
* ```javascript
* {
@@ -186,8 +186,8 @@ export class SystemStatus {
* ```
*
* Where the `isSystemIdle` property is set to `false` if the system
- * has been overloaded in the full history of the {@link Snapshotter}
- * (which is configurable in the {@link Snapshotter}) and `true` otherwise.
+ * has been overloaded in the full history of the {@apilink Snapshotter}
+ * (which is configurable in the {@apilink Snapshotter}) and `true` otherwise.
*/
getHistoricalStatus(): SystemInfo {
return this._isSystemIdle();
diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.ts
index 22b9aa656..459e3f741 100644
--- a/packages/core/src/configuration.ts
+++ b/packages/core/src/configuration.ts
@@ -13,19 +13,19 @@ import { LocalEventManager } from './events';
export interface ConfigurationOptions {
/**
* Defines storage client to be used.
- * @default {@link MemoryStorage}
+ * @default {@apilink MemoryStorage}
*/
storageClient?: StorageClient;
/**
* Defines the Event Manager to be used.
- * @default {@link EventManager}
+ * @default {@apilink EventManager}
*/
eventManager?: EventManager;
/**
* Could be used to adjust the storage client behavior
- * e.g. {@link MemoryStorageOptions} could be used to adjust the {@link MemoryStorage} behavior.
+ * e.g. {@apilink MemoryStorageOptions} could be used to adjust the {@apilink MemoryStorage} behavior.
*/
storageClientOptions?: Dictionary;
@@ -69,7 +69,7 @@ export interface ConfigurationOptions {
maxUsedCpuRatio?: number;
/**
- * Sets the ratio, defining the amount of system memory that could be used by the {@link AutoscaledPool}.
+ * Sets the ratio, defining the amount of system memory that could be used by the {@apilink AutoscaledPool}.
* When the memory usage is more than the provided ratio, the memory is considered overloaded.
*
* Alternative to `CRAWLEE_AVAILABLE_MEMORY_RATIO` environment variable.
@@ -78,7 +78,7 @@ export interface ConfigurationOptions {
availableMemoryRatio?: number;
/**
- * Sets the amount of system memory in megabytes to be used by the {@link AutoscaledPool}.
+ * Sets the amount of system memory in megabytes to be used by the {@apilink AutoscaledPool}.
* By default, the maximum memory is set to one quarter of total system memory.
*
* Alternative to `CRAWLEE_MEMORY_MBYTES` environment variable.
@@ -101,7 +101,7 @@ export interface ConfigurationOptions {
/**
* Defines the default input key, i.e. the key that is used to get the crawler input value
- * from the default {@link KeyValueStore} associated with the current crawler run.
+ * from the default {@apilink KeyValueStore} associated with the current crawler run.
*
* Alternative to `CRAWLEE_INPUT_KEY` environment variable.
* @default 'INPUT'
@@ -337,8 +337,8 @@ export class Configuration {
}
/**
- * Returns cached instance of {@link StorageClient} using options as defined in the environment variables or in
- * this {@link Configuration} instance. Only first call of this method will create the client, following calls will
+ * Returns cached instance of {@apilink StorageClient} using options as defined in the environment variables or in
+ * this {@apilink Configuration} instance. Only first call of this method will create the client, following calls will
* return the same client instance.
*
* Caching works based on the `storageClientOptions`, so calling this method with different options will return
@@ -407,14 +407,14 @@ export class Configuration {
}
/**
- * Gets default {@link StorageClient} instance.
+ * Gets default {@apilink StorageClient} instance.
*/
static getStorageClient(): StorageClient {
return this.getGlobalConfig().getStorageClient();
}
/**
- * Gets default {@link EventManager} instance.
+ * Gets default {@apilink EventManager} instance.
*/
static getEventManager(): EventManager {
return this.getGlobalConfig().getEventManager();
diff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.ts
index 03306bc12..2fbf81f9d 100644
--- a/packages/core/src/crawlers/crawler_commons.ts
+++ b/packages/core/src/crawlers/crawler_commons.ts
@@ -7,14 +7,14 @@ import type { Session } from '../session_pool/session';
export interface CrawlingContext<UserData extends Dictionary = Dictionary> extends Record<PropertyKey, unknown> {
id: string;
/**
- * The original {@link Request} object.
+ * The original {@apilink Request} object.
*/
request: Request<UserData>;
session?: Session;
/**
* An object with information about currently used proxy by the crawler
- * and configured by the {@link ProxyConfiguration} class.
+ * and configured by the {@apilink ProxyConfiguration} class.
*/
proxyInfo?: ProxyInfo;
log: Log;
diff --git a/packages/core/src/crawlers/statistics.ts b/packages/core/src/crawlers/statistics.ts
index 3c63c2055..ed2a81139 100644
--- a/packages/core/src/crawlers/statistics.ts
+++ b/packages/core/src/crawlers/statistics.ts
@@ -47,7 +47,7 @@ export class Statistics {
readonly id = Statistics.id++; // assign an id while incrementing so it can be saved/restored from KV
/**
- * Current statistic state used for doing calculations on {@link Statistics.calculate} calls
+ * Current statistic state used for doing calculations on {@apilink Statistics.calculate} calls
*/
state!: StatisticState;
diff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/core/src/enqueue_links/enqueue_links.ts
index 17a9c61fd..4fd1b89bf 100644
--- a/packages/core/src/enqueue_links/enqueue_links.ts
+++ b/packages/core/src/enqueue_links/enqueue_links.ts
@@ -26,10 +26,10 @@ export interface EnqueueLinksOptions {
/** A CSS selector matching links to be enqueued. */
selector?: string;
- /** Sets {@link Request.userData} for newly enqueued requests. */
+ /** Sets {@apilink Request.userData} for newly enqueued requests. */
userData?: Dictionary;
- /** Sets {@link Request.label} for newly enqueued requests. */
+ /** Sets {@apilink Request.label} for newly enqueued requests. */
label?: string;
/**
@@ -43,7 +43,7 @@ export interface EnqueueLinksOptions {
* containing glob pattern strings matching the URLs to be enqueued.
*
* The plain objects must include at least the `glob` property, which holds the glob pattern string.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* The matching is always case-insensitive.
* If you need case-sensitive matching, use `regexps` property directly.
@@ -58,7 +58,7 @@ export interface EnqueueLinksOptions {
* containing regular expressions matching the URLs to be enqueued.
*
* The plain objects must include at least the `regexp` property, which holds the regular expression.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* If `regexps` is an empty array or `undefined`, and `globs` are also not defined, then the function
* enqueues the links with the same subdomain.
@@ -69,11 +69,11 @@ export interface EnqueueLinksOptions {
* *NOTE:* In future versions of SDK the options will be removed.
* Please use `globs` or `regexps` instead.
*
- * An array of {@link PseudoUrl} strings or plain objects
- * containing {@link PseudoUrl} strings matching the URLs to be enqueued.
+ * An array of {@apilink PseudoUrl} strings or plain objects
+ * containing {@apilink PseudoUrl} strings matching the URLs to be enqueued.
*
* The plain objects must include at least the `purl` property, which holds the pseudo-URL string.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* With a pseudo-URL string, the matching is always case-insensitive.
* If you need case-sensitive matching, use `regexps` property directly.
@@ -86,7 +86,7 @@ export interface EnqueueLinksOptions {
pseudoUrls?: PseudoUrlInput[];
/**
- * Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
+ * Just before a new {@apilink Request} is constructed and enqueued to the {@apilink RequestQueue}, this function can be used
* to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
* when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
* or to dynamically update or create `userData`.
@@ -140,11 +140,11 @@ export enum EnqueueStrategy {
}
/**
- * This function enqueues the urls provided to the {@link RequestQueue} provided. If you want to automatically find and enqueue links,
+ * This function enqueues the urls provided to the {@apilink RequestQueue} provided. If you want to automatically find and enqueue links,
* you should use the context-aware `enqueueLinks` function provided on the crawler contexts.
*
* Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions
- * and override settings of the enqueued {@link Request} objects.
+ * and override settings of the enqueued {@apilink Request} objects.
*
* **Example usage**
*
@@ -161,7 +161,7 @@ export enum EnqueueStrategy {
* ```
*
* @param options All `enqueueLinks()` parameters are passed via an options object.
- * @returns Promise that resolves to {@link BatchAddRequestsResult} object.
+ * @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
export async function enqueueLinks(options: EnqueueLinksOptions): Promise<BatchAddRequestsResult> {
ow(options, ow.object.exactShape({
@@ -270,9 +270,9 @@ export async function enqueueLinks(options: EnqueueLinksOptions): Promise<BatchA
/**
* @internal
- * This method helps resolve the baseUrl that will be used for filtering in {@link enqueueLinks}.
+ * This method helps resolve the baseUrl that will be used for filtering in {@apilink enqueueLinks}.
* - If a user provides a base url, we always return it
- * - If a user specifies {@link EnqueueStrategy.All} strategy, they do not care if the newly found urls are on the original request domain, or a redirected one
+ * - If a user specifies {@apilink EnqueueStrategy.All} strategy, they do not care if the newly found urls are on the original request domain, or a redirected one
* - In all other cases, we return the domain of the original request as that's the one we need to use for filtering
*/
export function resolveBaseUrlForEnqueueLinksFiltering({
diff --git a/packages/core/src/enqueue_links/shared.ts b/packages/core/src/enqueue_links/shared.ts
index d5ca928ec..e9132e752 100644
--- a/packages/core/src/enqueue_links/shared.ts
+++ b/packages/core/src/enqueue_links/shared.ts
@@ -177,8 +177,8 @@ export function createRequestOptions(
}
/**
- * Takes an Apify {@link RequestOptions} object and changes its attributes in a desired way. This user-function is used
- * {@link enqueueLinks} to modify requests before enqueuing them.
+ * Takes an Apify {@apilink RequestOptions} object and changes its attributes in a desired way. This user-function is used
+ * {@apilink enqueueLinks} to modify requests before enqueuing them.
*/
export interface RequestTransform {
/**
diff --git a/packages/core/src/proxy_configuration.ts b/packages/core/src/proxy_configuration.ts
index 05242035a..9c1088d6f 100644
--- a/packages/core/src/proxy_configuration.ts
+++ b/packages/core/src/proxy_configuration.ts
@@ -17,7 +17,7 @@ export interface ProxyConfigurationOptions {
/**
* Custom function that allows you to generate the new proxy URL dynamically. It gets the `sessionId` as a parameter
* and should always return stringified proxy URL. Can be asynchronous.
- * This function is used to generate the URL when {@link ProxyConfiguration.newUrl} or {@link ProxyConfiguration.newProxyInfo} is called.
+ * This function is used to generate the URL when {@apilink ProxyConfiguration.newUrl} or {@apilink ProxyConfiguration.newProxyInfo} is called.
*/
newUrlFunction?: ProxyConfigurationFunction;
}
@@ -25,7 +25,7 @@ export interface ProxyConfigurationOptions {
/**
* The main purpose of the ProxyInfo object is to provide information
* about the current proxy connection used by the crawler for the request.
- * Outside of crawlers, you can get this object by calling {@link ProxyConfiguration.newProxyInfo}.
+ * Outside of crawlers, you can get this object by calling {@apilink ProxyConfiguration.newProxyInfo}.
*
* **Example usage:**
*
@@ -54,7 +54,7 @@ export interface ProxyConfigurationOptions {
*/
export interface ProxyInfo {
/**
- * The identifier of used {@link Session}, if used.
+ * The identifier of used {@apilink Session}, if used.
*/
sessionId?: string;
@@ -88,9 +88,9 @@ export interface ProxyInfo {
* Configures connection to a proxy server with the provided options. Proxy servers are used to prevent target websites from blocking
* your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures
* them to use the selected proxies for all connections. You can get information about the currently used proxy by inspecting
- * the {@link ProxyInfo} property in your crawler's page function. There, you can inspect the proxy's URL and other attributes.
+ * the {@apilink ProxyInfo} property in your crawler's page function. There, you can inspect the proxy's URL and other attributes.
*
- * If you want to use your own proxies, use the {@link ProxyConfigurationOptions.proxyUrls} option. Your list of proxy URLs will
+ * If you want to use your own proxies, use the {@apilink ProxyConfigurationOptions.proxyUrls} option. Your list of proxy URLs will
* be rotated by the configuration if this option is provided.
*
* **Example usage:**
@@ -121,7 +121,7 @@ export class ProxyConfiguration {
protected log = log.child({ prefix: 'ProxyConfiguration' });
/**
- * Creates a {@link ProxyConfiguration} instance based on the provided options. Proxy servers are used to prevent target websites from
+ * Creates a {@apilink ProxyConfiguration} instance based on the provided options. Proxy servers are used to prevent target websites from
* blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures
* them to use the selected proxies for all connections.
*
@@ -157,16 +157,16 @@ export class ProxyConfiguration {
}
/**
- * This function creates a new {@link ProxyInfo} info object.
+ * This function creates a new {@apilink ProxyInfo} info object.
* It is used by CheerioCrawler and PuppeteerCrawler to generate proxy URLs and also to allow the user to inspect
* the currently used proxy via the requestHandler parameter `proxyInfo`.
* Use it if you want to work with a rich representation of a proxy URL.
- * If you need the URL string only, use {@link ProxyConfiguration.newUrl}.
+ * If you need the URL string only, use {@apilink ProxyConfiguration.newUrl}.
* @param [sessionId]
- * Represents the identifier of user {@link Session} that can be managed by the {@link SessionPool} or
+ * Represents the identifier of user {@apilink Session} that can be managed by the {@apilink SessionPool} or
* you can use the Apify Proxy [Session](https://docs.apify.com/proxy#sessions) identifier.
* When the provided sessionId is a number, it's converted to a string. Property sessionId of
- * {@link ProxyInfo} is always returned as a type string.
+ * {@apilink ProxyInfo} is always returned as a type string.
*
* All the HTTP requests going through the proxy with the same session identifier
* will use the same target proxy server (i.e. the same IP address).
@@ -192,7 +192,7 @@ export class ProxyConfiguration {
/**
* Returns a new proxy URL based on provided configuration options and the `sessionId` parameter.
* @param [sessionId]
- * Represents the identifier of user {@link Session} that can be managed by the {@link SessionPool} or
+ * Represents the identifier of user {@apilink Session} that can be managed by the {@apilink SessionPool} or
* you can use the Apify Proxy [Session](https://docs.apify.com/proxy#sessions) identifier.
* When the provided sessionId is a number, it's converted to a string.
*
diff --git a/packages/core/src/request.ts b/packages/core/src/request.ts
index 0ae747896..ea48600fa 100644
--- a/packages/core/src/request.ts
+++ b/packages/core/src/request.ts
@@ -37,7 +37,7 @@ const requestOptionalPredicates = {
* Each `Request` instance has the `uniqueKey` property, which can be either specified
* manually in the constructor or generated automatically from the URL. Two requests with the same `uniqueKey`
* are considered as pointing to the same web resource. This behavior applies to all Apify SDK classes,
- * such as {@link RequestList}, {@link RequestQueue}, {@link PuppeteerCrawler} or {@link PlaywrightCrawler}.
+ * such as {@apilink RequestList}, {@apilink RequestQueue}, {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler}.
*
* Example use:
*
@@ -69,7 +69,7 @@ export class Request<UserData extends Dictionary = Dictionary> {
* An actually loaded URL after redirects, if present. HTTP redirects are guaranteed
* to be included.
*
- * When using {@link PuppeteerCrawler} or {@link PlaywrightCrawler}, meta tag and JavaScript redirects may,
+ * When using {@apilink PuppeteerCrawler} or {@apilink PlaywrightCrawler}, meta tag and JavaScript redirects may,
* or may not be included, depending on their nature. This generally means that redirects,
* which happen immediately will most likely be included, but delayed redirects will not.
*/
@@ -315,7 +315,7 @@ export class Request<UserData extends Dictionary = Dictionary> {
}
/**
- * Specifies required and optional fields for constructing a {@link Request}.
+ * Specifies required and optional fields for constructing a {@apilink Request}.
*/
export interface RequestOptions<UserData extends Dictionary = Dictionary> {
diff --git a/packages/core/src/session_pool/session.ts b/packages/core/src/session_pool/session.ts
index 1fe984787..0809f69e4 100644
--- a/packages/core/src/session_pool/session.ts
+++ b/packages/core/src/session_pool/session.ts
@@ -11,7 +11,7 @@ import { EVENT_SESSION_RETIRED } from './events';
import { browserPoolCookieToToughCookie, getCookiesFromResponse, getDefaultCookieExpirationDate, toughCookieToBrowserPoolCookie } from '../cookie_utils';
/**
- * Persistable {@link Session} state.
+ * Persistable {@apilink Session} state.
*/
export interface SessionState {
id: string;
diff --git a/packages/core/src/session_pool/session_pool.ts b/packages/core/src/session_pool/session_pool.ts
index e3e02cfee..6c831e080 100644
--- a/packages/core/src/session_pool/session_pool.ts
+++ b/packages/core/src/session_pool/session_pool.ts
@@ -11,7 +11,7 @@ import type { EventManager } from '../events/event_manager';
import { EventType } from '../events/event_manager';
/**
- * Factory user-function which creates customized {@link Session} instances.
+ * Factory user-function which creates customized {@apilink Session} instances.
*/
export interface CreateSession {
/**
@@ -28,7 +28,7 @@ export interface SessionPoolOptions {
*/
maxPoolSize?: number;
- /** The configuration options for {@link Session} instances. */
+ /** The configuration options for {@apilink Session} instances. */
sessionOptions?: SessionOptions;
/** Name or Id of `KeyValueStore` where is the `SessionPool` state stored. */
@@ -60,11 +60,11 @@ export interface SessionPoolOptions {
/**
* Handles the rotation, creation and persistence of user-like sessions.
- * Creates a pool of {@link Session} instances, that are randomly rotated.
+ * Creates a pool of {@apilink Session} instances, that are randomly rotated.
* When some session is marked as blocked, it is removed and new one is created instead (the pool never returns an unusable session).
* Learn more in the [Session management guide](../../../docs/guides/session-management).
*
- * You can create one by calling the {@link SessionPool.open} function.
+ * You can create one by calling the {@apilink SessionPool.open} function.
*
* Session pool is already integrated into crawlers, and it can significantly improve your scraper
* performance with just 2 lines of code.
@@ -79,10 +79,10 @@ export interface SessionPoolOptions {
* })
* ```
*
- * You can configure the pool with many options. See the {@link SessionPoolOptions}.
- * Session pool is by default persisted in default {@link KeyValueStore}.
+ * You can configure the pool with many options. See the {@apilink SessionPoolOptions}.
+ * Session pool is by default persisted in default {@apilink KeyValueStore}.
* If you want to have one pool for all runs you have to specify
- * {@link SessionPoolOptions.persistStateKeyValueStoreId}.
+ * {@apilink SessionPoolOptions.persistStateKeyValueStoreId}.
*
* **Advanced usage:**
*
@@ -193,8 +193,8 @@ export class SessionPool extends EventEmitter {
}
/**
- * Starts periodic state persistence and potentially loads SessionPool state from {@link KeyValueStore}.
- * It is called automatically by the {@link SessionPool.open} function.
+ * Starts periodic state persistence and potentially loads SessionPool state from {@apilink KeyValueStore}.
+ * It is called automatically by the {@apilink SessionPool.open} function.
*/
async initialize(): Promise<void> {
this.keyValueStore = await KeyValueStore.open(this.persistStateKeyValueStoreId, { config: this.config });
@@ -294,7 +294,7 @@ export class SessionPool extends EventEmitter {
}
/**
- * Persists the current state of the `SessionPool` into the default {@link KeyValueStore}.
+ * Persists the current state of the `SessionPool` into the default {@apilink KeyValueStore}.
* The state is persisted automatically in regular intervals.
*/
async persistState(): Promise<void> {
@@ -426,9 +426,9 @@ export class SessionPool extends EventEmitter {
/**
* Opens a SessionPool and returns a promise resolving to an instance
- * of the {@link SessionPool} class that is already initialized.
+ * of the {@apilink SessionPool} class that is already initialized.
*
- * For more details and code examples, see the {@link SessionPool} class.
+ * For more details and code examples, see the {@apilink SessionPool} class.
*/
static async open(options?: SessionPoolOptions): Promise<SessionPool> {
const sessionPool = new SessionPool(options);
diff --git a/packages/core/src/storages/dataset.ts b/packages/core/src/storages/dataset.ts
index ee7287fe9..81092e3d2 100644
--- a/packages/core/src/storages/dataset.ts
+++ b/packages/core/src/storages/dataset.ts
@@ -167,7 +167,7 @@ export interface DatasetIteratorOptions extends Omit<DatasetDataOptions, 'offset
* Typically it is used to store crawling results.
*
* Do not instantiate this class directly, use the
- * {@link Dataset.open} function instead.
+ * {@apilink Dataset.open} function instead.
*
* `Dataset` stores its data either on local disk or in the Apify cloud,
* depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables are set.
@@ -184,7 +184,7 @@ export interface DatasetIteratorOptions extends Omit<DatasetDataOptions, 'offset
* If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the
* [Apify Dataset](https://docs.apify.com/storage/dataset)
* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
- * option to {@link Dataset.open} function,
+ * option to {@apilink Dataset.open} function,
* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.
*
* **Example usage:**
@@ -268,7 +268,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
}
/**
- * Returns {@link DatasetContent} object holding the items in the dataset based on the provided parameters.
+ * Returns {@apilink DatasetContent} object holding the items in the dataset based on the provided parameters.
*/
async getData(options: DatasetDataOptions = {}): Promise<PaginatedList<Data>> {
try {
@@ -412,13 +412,13 @@ export class Dataset<Data extends Dictionary = Dictionary> {
}
/**
- * Opens a dataset and returns a promise resolving to an instance of the {@link Dataset} class.
+ * Opens a dataset and returns a promise resolving to an instance of the {@apilink Dataset} class.
*
* Datasets are used to store structured data where each object stored has the same attributes,
* such as online store products or real estate offers.
* The actual data is stored either on the local filesystem or in the cloud.
*
- * For more details and code examples, see the {@link Dataset} class.
+ * For more details and code examples, see the {@apilink Dataset} class.
*
* @param [datasetIdOrName]
* ID or name of the dataset to be opened. If `null` or `undefined`,
@@ -438,9 +438,9 @@ export class Dataset<Data extends Dictionary = Dictionary> {
}
/**
- * Stores an object or an array of objects to the default {@link Dataset} of the current crawler run.
+ * Stores an object or an array of objects to the default {@apilink Dataset} of the current crawler run.
*
- * This is just a convenient shortcut for {@link Dataset.pushData}.
+ * This is just a convenient shortcut for {@apilink Dataset.pushData}.
* For example, calling the following code:
* ```javascript
* await Dataset.pushData({ myValue: 123 });
@@ -452,7 +452,7 @@ export class Dataset<Data extends Dictionary = Dictionary> {
* await dataset.pushData({ myValue: 123 });
* ```
*
- * For more information, see {@link Dataset.open} and {@link Dataset.pushData}
+ * For more information, see {@apilink Dataset.open} and {@apilink Dataset.pushData}
*
* **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,
* otherwise the crawler process might finish before the data are stored!
@@ -473,8 +473,8 @@ export class Dataset<Data extends Dictionary = Dictionary> {
export interface DatasetConsumer<Data> {
/**
- * @param item Current {@link Dataset} entry being processed.
- * @param index Position of current {@link Dataset} entry.
+ * @param item Current {@apilink Dataset} entry being processed.
+ * @param index Position of current {@apilink Dataset} entry.
*/
(item: Data, index: number): Awaitable<void>;
@@ -487,8 +487,8 @@ export interface DatasetMapper<Data, R> {
/**
* User-function used in the `Dataset.map()` API.
- * @param item Current {@link Dataset} entry being processed.
- * @param index Position of current {@link Dataset} entry.
+ * @param item Current {@apilink Dataset} entry being processed.
+ * @param index Position of current {@apilink Dataset} entry.
*/
(item: Data, index: number): Awaitable<R>;
@@ -501,8 +501,8 @@ export interface DatasetReducer<T, Data> {
/**
* @param memo Previous state of the reduction.
- * @param item Current {@link Dataset} entry being processed.
- * @param index Position of current {@link Dataset} entry.
+ * @param item Current {@apilink Dataset} entry being processed.
+ * @param index Position of current {@apilink Dataset} entry.
*/
(memo: T, item: Data, index: number): Awaitable<T>;
diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.ts
index 85a863c76..89f829f09 100644
--- a/packages/core/src/storages/key_value_store.ts
+++ b/packages/core/src/storages/key_value_store.ts
@@ -46,16 +46,16 @@ export const maybeStringify = <T>(value: T, options: { contentType?: string }) =
* for saving screenshots, crawler inputs and outputs, web pages, PDFs or to persist the state of crawlers.
*
* Do not instantiate this class directly, use the
- * {@link KeyValueStore.open} function instead.
+ * {@apilink KeyValueStore.open} function instead.
*
* Each crawler run is associated with a default key-value store, which is created exclusively
* for the run. By convention, the crawler input and output are stored into the
* default key-value store under the `INPUT` and `OUTPUT` key, respectively.
* Typically, input and output are JSON files, although it can be any other format.
* To access the default key-value store directly, you can use the
- * {@link KeyValueStore.getValue} and {@link KeyValueStore.setValue} convenience functions.
+ * {@apilink KeyValueStore.getValue} and {@apilink KeyValueStore.setValue} convenience functions.
*
- * To access the input, you can also use the {@link KeyValueStore.getInput} convenience function.
+ * To access the input, you can also use the {@apilink KeyValueStore.getInput} convenience function.
*
* `KeyValueStore` stores its data on a local disk.
*
@@ -127,7 +127,7 @@ export class KeyValueStore {
* If the record does not exist, the function resolves to `null`.
*
* To save or delete a value in the key-value store, use the
- * {@link KeyValueStore.setValue} function.
+ * {@apilink KeyValueStore.setValue} function.
*
* **Example usage:**
*
@@ -205,7 +205,7 @@ export class KeyValueStore {
* regardless whether the record existed or not.
*
* To retrieve a value from the key-value store, use the
- * {@link KeyValueStore.getValue} function.
+ * {@apilink KeyValueStore.getValue} function.
*
* **IMPORTANT:** Always make sure to use the `await` keyword when calling `setValue()`,
* otherwise the crawler process might finish before the value is stored!
@@ -325,13 +325,13 @@ export class KeyValueStore {
}
/**
- * Opens a key-value store and returns a promise resolving to an instance of the {@link KeyValueStore} class.
+ * Opens a key-value store and returns a promise resolving to an instance of the {@apilink KeyValueStore} class.
*
* Key-value stores are used to store records or files, along with their MIME content type.
* The records are stored and retrieved using a unique key.
* The actual data is stored either on a local filesystem or in the Apify cloud.
*
- * For more details and code examples, see the {@link KeyValueStore} class.
+ * For more details and code examples, see the {@apilink KeyValueStore} class.
*
* @param [storeIdOrName]
* ID or name of the key-value store to be opened. If `null` or `undefined`,
@@ -350,9 +350,9 @@ export class KeyValueStore {
}
/**
- * Gets a value from the default {@link KeyValueStore} associated with the current crawler run.
+ * Gets a value from the default {@apilink KeyValueStore} associated with the current crawler run.
*
- * This is just a convenient shortcut for {@link KeyValueStore.getValue}.
+ * This is just a convenient shortcut for {@apilink KeyValueStore.getValue}.
* For example, calling the following code:
* ```javascript
* const value = await KeyValueStore.getValue('my-key');
@@ -364,10 +364,10 @@ export class KeyValueStore {
* const value = await store.getValue('my-key');
* ```
*
- * To store the value to the default key-value store, you can use the {@link KeyValueStore.setValue} function.
+ * To store the value to the default key-value store, you can use the {@apilink KeyValueStore.setValue} function.
*
- * For more information, see {@link KeyValueStore.open}
- * and {@link KeyValueStore.getValue}.
+ * For more information, see {@apilink KeyValueStore.open}
+ * and {@apilink KeyValueStore.getValue}.
*
* @param key Unique record key.
* @param defaultValue Fallback that will be returned if no value if present in the storage.
@@ -389,9 +389,9 @@ export class KeyValueStore {
}
/**
- * Stores or deletes a value in the default {@link KeyValueStore} associated with the current crawler run.
+ * Stores or deletes a value in the default {@apilink KeyValueStore} associated with the current crawler run.
*
- * This is just a convenient shortcut for {@link KeyValueStore.setValue}.
+ * This is just a convenient shortcut for {@apilink KeyValueStore.setValue}.
* For example, calling the following code:
* ```javascript
* await KeyValueStore.setValue('OUTPUT', { foo: "bar" });
@@ -403,10 +403,10 @@ export class KeyValueStore {
* await store.setValue('OUTPUT', { foo: "bar" });
* ```
*
- * To get a value from the default key-value store, you can use the {@link KeyValueStore.getValue} function.
+ * To get a value from the default key-value store, you can use the {@apilink KeyValueStore.getValue} function.
*
- * For more information, see {@link KeyValueStore.open}
- * and {@link KeyValueStore.getValue}.
+ * For more information, see {@apilink KeyValueStore.open}
+ * and {@apilink KeyValueStore.getValue}.
*
* @param key
* Unique record key.
@@ -425,7 +425,7 @@ export class KeyValueStore {
}
/**
- * Gets the crawler input value from the default {@link KeyValueStore} associated with the current crawler run.
+ * Gets the crawler input value from the default {@apilink KeyValueStore} associated with the current crawler run.
*
* This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](core/class/KeyValueStore#getValue).
* For example, calling the following code:
@@ -443,8 +443,8 @@ export class KeyValueStore {
* If you need to use the input multiple times in your crawler,
* it is far more efficient to read it once and store it locally.
*
- * For more information, see {@link KeyValueStore.open}
- * and {@link KeyValueStore.getValue}.
+ * For more information, see {@apilink KeyValueStore.open}
+ * and {@apilink KeyValueStore.getValue}.
*
* @returns
* Returns a promise that resolves to an object, string
@@ -460,13 +460,13 @@ export class KeyValueStore {
}
/**
- * User-function used in the {@link KeyValueStore.forEachKey} method.
+ * User-function used in the {@apilink KeyValueStore.forEachKey} method.
*/
export interface KeyConsumer {
/**
- * @param key Current {@link KeyValueStore} key being processed.
- * @param index Position of the current key in {@link KeyValueStore}.
- * @param info Information about the current {@link KeyValueStore} entry.
+ * @param key Current {@apilink KeyValueStore} key being processed.
+ * @param index Position of the current key in {@apilink KeyValueStore}.
+ * @param info Information about the current {@apilink KeyValueStore} entry.
* @param info.size Size of the value associated with the current key in bytes.
*/
(key: string, index: number, info: { size: number }): Awaitable<void>;
diff --git a/packages/core/src/storages/request_list.ts b/packages/core/src/storages/request_list.ts
index 033f7a7fa..4b690ad0d 100644
--- a/packages/core/src/storages/request_list.ts
+++ b/packages/core/src/storages/request_list.ts
@@ -22,8 +22,8 @@ const CONTENT_TYPE_BINARY = 'application/octet-stream';
export interface RequestListOptions {
/**
- * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,
- * plain objects that define at least the `url` property, or an array of {@link Request} instances.
+ * An array of sources of URLs for the {@apilink RequestList}. It can be either an array of strings,
+ * plain objects that define at least the `url` property, or an array of {@apilink Request} instances.
*
* **IMPORTANT:** The `sources` array will be consumed (left empty) after `RequestList` initializes.
* This is a measure to prevent memory leaks in situations when millions of sources are
@@ -58,15 +58,15 @@ export interface RequestListOptions {
/**
* A function that will be called to get the sources for the `RequestList`, but only if `RequestList`
- * was not able to fetch their persisted version (see {@link RequestListOptions.persistRequestsKey}).
- * It must return an `Array` of {@link Request} or {@link RequestOptions}.
+ * was not able to fetch their persisted version (see {@apilink RequestListOptions.persistRequestsKey}).
+ * It must return an `Array` of {@apilink Request} or {@apilink RequestOptions}.
*
* This is very useful in a scenario when getting the sources is a resource intensive or time consuming
* task, such as fetching URLs from multiple sitemaps or parsing URLs from large datasets. Using the
* `sourcesFunction` in combination with `persistStateKey` and `persistRequestsKey` will allow you to
* fetch and parse those URLs only once, saving valuable time when your crawler migrates or restarts.
*
- * If both {@link RequestListOptions.sources} and {@link RequestListOptions.sourcesFunction} are provided,
+ * If both {@apilink RequestListOptions.sources} and {@apilink RequestListOptions.sourcesFunction} are provided,
* the sources returned by the function will be added after the `sources`.
*
* **Example:**
@@ -116,12 +116,12 @@ export interface RequestListOptions {
/**
* Identifies the key in the default key-value store under which the `RequestList` persists its
- * Requests during the {@link RequestList.initialize} call.
+ * Requests during the {@apilink RequestList.initialize} call.
* This is necessary if `persistStateKey` is set and the source URLs might potentially change,
* to ensure consistency of the source URLs and state object. However, it comes with some
* storage and performance overheads.
*
- * If `persistRequestsKey` is not set, {@link RequestList.initialize} will always fetch the sources
+ * If `persistRequestsKey` is not set, {@apilink RequestList.initialize} will always fetch the sources
* from their origin, check that they are consistent with the restored state (if any)
* and throw an error if they are not.
*/
@@ -149,7 +149,7 @@ export interface RequestListOptions {
/**
* By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
- * on the `uniqueKey` property of passed source {@link Request} objects.
+ * on the `uniqueKey` property of passed source {@apilink Request} objects.
*
* If the property is not present, it is generated by normalizing the URL. If present, it is kept intact.
* In any case, only one request per `uniqueKey` is added to the `RequestList` resulting in removal
@@ -171,19 +171,19 @@ export interface RequestListOptions {
/**
* Represents a static list of URLs to crawl.
* The URLs can be provided either in code or parsed from a text file hosted on the web.
- * `RequestList` is used by {@link BasicCrawler}, {@link CheerioCrawler}, {@link PuppeteerCrawler}
- * and {@link PlaywrightCrawler} as a source of URLs to crawl.
+ * `RequestList` is used by {@apilink BasicCrawler}, {@apilink CheerioCrawler}, {@apilink PuppeteerCrawler}
+ * and {@apilink PlaywrightCrawler} as a source of URLs to crawl.
*
- * Each URL is represented using an instance of the {@link Request} class.
+ * Each URL is represented using an instance of the {@apilink Request} class.
* The list can only contain unique URLs. More precisely, it can only contain `Request` instances
* with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
- * To add a single URL to the list multiple times, corresponding {@link Request} objects will need to have different
+ * To add a single URL to the list multiple times, corresponding {@apilink Request} objects will need to have different
* `uniqueKey` properties. You can use the `keepDuplicateUrls` option to do this for you when initializing the
* `RequestList` from sources.
*
- * Once you create an instance of `RequestList`, you need to call the {@link RequestList.initialize} function
+ * Once you create an instance of `RequestList`, you need to call the {@apilink RequestList.initialize} function
* before the instance can be used. After that, no more URLs can be added to the list.
- * Unlike {@link RequestQueue}, `RequestList` is static but it can contain even millions of URLs.
+ * Unlike {@apilink RequestQueue}, `RequestList` is static but it can contain even millions of URLs.
* > Note that `RequestList` can be used together with `RequestQueue` by the same crawler.
* > In such cases, each request from `RequestList` is enqueued into `RequestQueue` first and then consumed from the latter.
* > This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue).
@@ -192,9 +192,9 @@ export interface RequestListOptions {
*
* `RequestList` has an internal state where it stores information about which requests were already handled,
* which are in progress and which were reclaimed. The state may be automatically persisted to the default
- * {@link KeyValueStore} by setting the `persistStateKey` option so that if the Node.js process is restarted,
+ * {@apilink KeyValueStore} by setting the `persistStateKey` option so that if the Node.js process is restarted,
* the crawling can continue where it left off. The automated persisting is launched upon receiving the `persistState`
- * event that is periodically emitted by {@link EventManager}.
+ * event that is periodically emitted by {@apilink EventManager}.
*
* The internal state is closely tied to the provided sources (URLs). If the sources change on crawler restart, the state will become corrupted and
* `RequestList` will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web.
@@ -428,7 +428,7 @@ export class RequestList {
}
/**
- * Persists the current state of the `RequestList` into the default {@link KeyValueStore}.
+ * Persists the current state of the `RequestList` into the default {@apilink KeyValueStore}.
* The state is persisted automatically in regular intervals, but calling this method manually
* is useful in cases where you want to have the most current state available after you pause
* or stop fetching its requests. For example after you pause or abort a crawl. Or just before
@@ -560,7 +560,7 @@ export class RequestList {
}
/**
- * Resolves to `true` if the next call to {@link RequestList.fetchNextRequest} function
+ * Resolves to `true` if the next call to {@apilink RequestList.fetchNextRequest} function
* would return `null`, otherwise it resolves to `false`.
* Note that even if the list is empty, there might be some pending requests currently being processed.
*/
@@ -580,8 +580,8 @@ export class RequestList {
}
/**
- * Gets the next {@link Request} to process. First, the function gets a request previously reclaimed
- * using the {@link RequestList.reclaimRequest} function, if there is any.
+ * Gets the next {@apilink Request} to process. First, the function gets a request previously reclaimed
+ * using the {@apilink RequestList.reclaimRequest} function, if there is any.
* Otherwise it gets the next request from sources.
*
* The function's `Promise` resolves to `null` if there are no more
@@ -778,14 +778,14 @@ export class RequestList {
/**
* Opens a request list and returns a promise resolving to an instance
- * of the {@link RequestList} class that is already initialized.
+ * of the {@apilink RequestList} class that is already initialized.
*
- * {@link RequestList} represents a list of URLs to crawl, which is always stored in memory.
+ * {@apilink RequestList} represents a list of URLs to crawl, which is always stored in memory.
* To enable picking up where left off after a process restart, the request list sources
* are persisted to the key-value store at initialization of the list. Then, while crawling,
* a small state object is regularly persisted to keep track of the crawling status.
*
- * For more details and code examples, see the {@link RequestList} class.
+ * For more details and code examples, see the {@apilink RequestList} class.
*
* **Example usage:**
*
@@ -811,23 +811,23 @@ export class RequestList {
* If `null`, the list will not be persisted and will only be stored in memory. Process restart
* will then cause the list to be crawled again from the beginning. We suggest always using a name.
* @param [sources]
- * An array of sources of URLs for the {@link RequestList}. It can be either an array of strings,
- * plain objects that define at least the `url` property, or an array of {@link Request} instances.
+ * An array of sources of URLs for the {@apilink RequestList}. It can be either an array of strings,
+ * plain objects that define at least the `url` property, or an array of {@apilink Request} instances.
*
- * **IMPORTANT:** The `sources` array will be consumed (left empty) after {@link RequestList} initializes.
+ * **IMPORTANT:** The `sources` array will be consumed (left empty) after {@apilink RequestList} initializes.
* This is a measure to prevent memory leaks in situations when millions of sources are
* added.
*
* Additionally, the `requestsFromUrl` property may be used instead of `url`,
- * which will instruct {@link RequestList} to download the source URLs from a given remote location.
+ * which will instruct {@apilink RequestList} to download the source URLs from a given remote location.
* The URLs will be parsed from the received response. In this case you can limit the URLs
* using `regex` parameter containing regular expression pattern for URLs to be included.
*
- * For details, see the {@link RequestListOptions.sources}
+ * For details, see the {@apilink RequestListOptions.sources}
* @param [options]
- * The {@link RequestList} options. Note that the `listName` parameter supersedes
- * the {@link RequestListOptions.persistStateKey} and {@link RequestListOptions.persistRequestsKey}
- * options and the `sources` parameter supersedes the {@link RequestListOptions.sources} option.
+ * The {@apilink RequestList} options. Note that the `listName` parameter supersedes
+ * the {@apilink RequestListOptions.persistStateKey} and {@apilink RequestListOptions.persistRequestsKey}
+ * options and the `sources` parameter supersedes the {@apilink RequestListOptions.sources} option.
*/
static async open(listNameOrOptions: string | null | RequestListOptions, sources?: Source[], options: RequestListOptions = {}): Promise<RequestList> {
if (listNameOrOptions != null && typeof listNameOrOptions === 'object') {
@@ -864,8 +864,8 @@ export class RequestList {
}
/**
- * Represents state of a {@link RequestList}. It can be used to resume a {@link RequestList} which has been previously processed.
- * You can obtain the state by calling {@link RequestList.getState} and receive an object with
+ * Represents state of a {@apilink RequestList}. It can be used to resume a {@apilink RequestList} which has been previously processed.
+ * You can obtain the state by calling {@apilink RequestList.getState} and receive an object with
* the following structure:
*
* ```
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index 830e4ed67..181ad735c 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -72,7 +72,7 @@ export function getRequestId(uniqueKey: string) {
/**
* A helper class that is used to report results from various
- * {@link RequestQueue} functions as well as {@link enqueueLinks}.
+ * {@apilink RequestQueue} functions as well as {@apilink enqueueLinks}.
*/
export interface QueueOperationInfo {
@@ -94,7 +94,7 @@ export interface RequestQueueOperationOptions {
* If set to `true`:
* - while adding the request to the queue: the request will be added to the foremost position in the queue.
* - while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned
- * in the next call to {@link RequestQueue.fetchNextRequest}.
+ * in the next call to {@apilink RequestQueue.fetchNextRequest}.
* By default, it's put to the end of the queue.
* @default false
*/
@@ -106,17 +106,17 @@ export interface RequestQueueOperationOptions {
* where you start with several URLs and then recursively
* follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
*
- * Each URL is represented using an instance of the {@link Request} class.
- * The queue can only contain unique URLs. More precisely, it can only contain {@link Request} instances
+ * Each URL is represented using an instance of the {@apilink Request} class.
+ * The queue can only contain unique URLs. More precisely, it can only contain {@apilink Request} instances
* with distinct `uniqueKey` properties. By default, `uniqueKey` is generated from the URL, but it can also be overridden.
* To add a single URL multiple times to the queue,
- * corresponding {@link Request} objects will need to have different `uniqueKey` properties.
+ * corresponding {@apilink Request} objects will need to have different `uniqueKey` properties.
*
- * Do not instantiate this class directly, use the {@link RequestQueue.open} function instead.
+ * Do not instantiate this class directly, use the {@apilink RequestQueue.open} function instead.
*
- * `RequestQueue` is used by {@link BasicCrawler}, {@link CheerioCrawler}, {@link PuppeteerCrawler}
- * and {@link PlaywrightCrawler} as a source of URLs to crawl.
- * Unlike {@link RequestList}, `RequestQueue` supports dynamic adding and removing of requests.
+ * `RequestQueue` is used by {@apilink BasicCrawler}, {@apilink CheerioCrawler}, {@apilink PuppeteerCrawler}
+ * and {@apilink PlaywrightCrawler} as a source of URLs to crawl.
+ * Unlike {@apilink RequestList}, `RequestQueue` supports dynamic adding and removing of requests.
* On the other hand, the queue is not optimized for operations that add or remove a large number of URLs in a batch.
*
* `RequestQueue` stores its data either on local disk or in the Apify Cloud,
@@ -128,7 +128,7 @@ export interface RequestQueueOperationOptions {
* If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` is not, the data is stored in the
* [Apify Request Queue](https://docs.apify.com/storage/request-queue)
* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
- * option to {@link RequestQueue.open} function,
+ * option to {@apilink RequestQueue.open} function,
* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.
*
* **Example usage:**
@@ -219,12 +219,12 @@ export class RequestQueue {
*
* If a request with the same `uniqueKey` property is already present in the queue,
* it will not be updated. You can find out whether this happened from the resulting
- * {@link QueueOperationInfo} object.
+ * {@apilink QueueOperationInfo} object.
*
* To add multiple requests to the queue by extracting links from a webpage,
- * see the {@link enqueueLinks} helper function.
+ * see the {@apilink enqueueLinks} helper function.
*
- * @param requestLike {@link Request} object or vanilla object with request data.
+ * @param requestLike {@apilink Request} object or vanilla object with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed Request.
* @param [options] Request queue operation options.
*/
@@ -280,9 +280,9 @@ export class RequestQueue {
*
* If a request that is passed in is already present due to its `uniqueKey` property being the same,
* it will not be updated. You can find out whether this happened by finding the request in the resulting
- * {@link BatchAddRequestsResult} object.
+ * {@apilink BatchAddRequestsResult} object.
*
- * @param requestsLike {@link Request} objects or vanilla objects with request data.
+ * @param requestsLike {@apilink Request} objects or vanilla objects with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed requests if missing.
* @param [options] Request queue operation options.
*/
@@ -394,15 +394,15 @@ export class RequestQueue {
* Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
*
* Once you successfully finish processing of the request, you need to call
- * {@link RequestQueue.markRequestHandled}
+ * {@apilink RequestQueue.markRequestHandled}
* to mark the request as handled in the queue. If there was some error in processing the request,
- * call {@link RequestQueue.reclaimRequest} instead,
+ * call {@apilink RequestQueue.reclaimRequest} instead,
* so that the queue will give the request to some other consumer in another call to the `fetchNextRequest` function.
*
* Note that the `null` return value doesn't mean the queue processing finished,
* it means there are currently no pending requests.
* To check whether all requests in queue were finished,
- * use {@link RequestQueue.isFinished} instead.
+ * use {@apilink RequestQueue.isFinished} instead.
*
* @returns
* Returns the request object or `null` if there are no more pending requests.
@@ -467,7 +467,7 @@ export class RequestQueue {
/**
* Marks a request that was previously returned by the
- * {@link RequestQueue.fetchNextRequest}
+ * {@apilink RequestQueue.fetchNextRequest}
* function as handled after successful processing.
* Handled requests will never again be returned by the `fetchNextRequest` function.
*/
@@ -503,7 +503,7 @@ export class RequestQueue {
/**
* Reclaims a failed request back to the queue, so that it can be returned for processing later again
- * by another call to {@link RequestQueue.fetchNextRequest}.
+ * by another call to {@apilink RequestQueue.fetchNextRequest}.
* The request record in the queue is updated using the provided `request` parameter.
* For example, this lets you store the number of retries or error messages for the request.
*/
@@ -548,10 +548,10 @@ export class RequestQueue {
}
/**
- * Resolves to `true` if the next call to {@link RequestQueue.fetchNextRequest}
+ * Resolves to `true` if the next call to {@apilink RequestQueue.fetchNextRequest}
* would return `null`, otherwise it resolves to `false`.
* Note that even if the queue is empty, there might be some pending requests currently being processed.
- * If you need to ensure that there is no activity in the queue, use {@link RequestQueue.isFinished}.
+ * If you need to ensure that there is no activity in the queue, use {@apilink RequestQueue.isFinished}.
*/
async isEmpty(): Promise<boolean> {
await this._ensureHeadIsNonEmpty();
@@ -764,14 +764,14 @@ export class RequestQueue {
/**
* Opens a request queue and returns a promise resolving to an instance
- * of the {@link RequestQueue} class.
+ * of the {@apilink RequestQueue} class.
*
- * {@link RequestQueue} represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud.
+ * {@apilink RequestQueue} represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud.
* The queue is used for deep crawling of websites, where you start with several URLs and then
* recursively follow links to other pages. The data structure supports both breadth-first
* and depth-first crawling orders.
*
- * For more details and code examples, see the {@link RequestQueue} class.
+ * For more details and code examples, see the {@apilink RequestQueue} class.
*
* @param [queueIdOrName]
* ID or name of the request queue to be opened. If `null` or `undefined`,
diff --git a/packages/core/src/storages/utils.ts b/packages/core/src/storages/utils.ts
index 0576aa972..299a2ace8 100644
--- a/packages/core/src/storages/utils.ts
+++ b/packages/core/src/storages/utils.ts
@@ -6,7 +6,7 @@ import { Configuration } from '../configuration';
* Purging will remove all the files in all storages except for INPUT.json in the default KV store.
*
* Purging of storages is happening automatically when we run our crawler (or when we open some storage
- * explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@link Configuration}
+ * explicitly, e.g. via `RequestList.open()`). We can disable that via `purgeOnStart` {@apilink Configuration}
* option or by setting `CRAWLEE_PURGE_ON_START` environment variable to `0` or `false`.
*
* This is a shortcut for running (optional) `purge` method on the StorageClient interface, in other words
diff --git a/packages/playwright-crawler/README.md b/packages/playwright-crawler/README.md
new file mode 100644
index 000000000..858ee14fd
--- /dev/null
+++ b/packages/playwright-crawler/README.md
@@ -0,0 +1,47 @@
+# `@crawlee/playwright`
+
+Provides a simple framework for parallel crawling of web pages using headless Chromium, Firefox and Webkit browsers with [Playwright](https://github.com/microsoft/playwright). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+Since `Playwright` uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.
+
+The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) or [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) constructor options, respectively.
+
+If both [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) and [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+`PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl and then calls the function provided by user as the [PlaywrightCrawlerOptions.requestHandler](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestHandler) option.
+
+New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [PlaywrightCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#autoscaledPoolOptions) parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPoolOptions](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions) are available directly in the `PlaywrightCrawler` constructor.
+
+Note that the pool of Playwright instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.
+
+## Example usage
+
+```javascript
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ page, request }) {
+ // This function is called to extract data from a single web page
+ // 'page' is an instance of Playwright.Page with page.goto(request.url) already called
+ // 'request' is an instance of Request class with information about the page to load
+ await Dataset.pushData({
+ title: await page.title(),
+ url: request.url,
+ succeeded: true,
+ })
+ },
+ async failedRequestHandler({ request }) {
+ // This function is called when the crawling of a request failed too many times
+ await Dataset.pushData({
+ url: request.url,
+ succeeded: false,
+ errors: request.errorMessages,
+ })
+ },
+});
+
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+]);
+```
diff --git a/packages/playwright-crawler/src/internals/playwright-crawler.ts b/packages/playwright-crawler/src/internals/playwright-crawler.ts
index a19c67f7f..764d819c1 100644
--- a/packages/playwright-crawler/src/internals/playwright-crawler.ts
+++ b/packages/playwright-crawler/src/internals/playwright-crawler.ts
@@ -20,15 +20,15 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
{ browserPlugins: [PlaywrightPlugin] }
> {
/**
- * The same options as used by {@link launchPlaywright}.
+ * The same options as used by {@apilink launchPlaywright}.
*/
launchContext?: PlaywrightLaunchContext;
/**
* Function that is called to process each request.
*
- * The function receives the {@link PlaywrightCrawlingContext} as an argument, where:
- * - `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * The function receives the {@apilink PlaywrightCrawlingContext} as an argument, where:
+ * - `request` is an instance of the {@apilink Request} object with details about the URL to open, HTTP method etc.
* - `page` is an instance of the `Playwright`
* [`Page`](https://playwright.dev/docs/api/class-page)
* - `browserController` is an instance of the
@@ -46,15 +46,15 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage} function.
+ * {@apilink Request.pushErrorMessage} function.
*/
requestHandler?: PlaywrightRequestHandler;
/**
* Function that is called to process each request.
*
- * The function receives the {@link PlaywrightCrawlingContext} as an argument, where:
- * - `request` is an instance of the {@link Request} object with details about the URL to open, HTTP method etc.
+ * The function receives the {@apilink PlaywrightCrawlingContext} as an argument, where:
+ * - `request` is an instance of the {@apilink Request} object with details about the URL to open, HTTP method etc.
* - `page` is an instance of the `Playwright`
* [`Page`](https://playwright.dev/docs/api/class-page)
* - `browserController` is an instance of the
@@ -72,7 +72,7 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
* The exceptions are logged to the request using the
- * {@link Request.pushErrorMessage} function.
+ * {@apilink Request.pushErrorMessage} function.
*
* @deprecated `handlePageFunction` has been renamed to `requestHandler` and will be removed in a future version.
* @ignore
@@ -120,27 +120,27 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
*
* Since `Playwright` uses headless browser to download web pages and extract data,
* it is useful for crawling of websites that require to execute JavaScript.
- * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * If the target website doesn't need JavaScript, consider using {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the {@link PlaywrightCrawlerOptions.requestList}
- * or {@link PlaywrightCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from
+ * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink PlaywrightCrawlerOptions.requestList}
+ * or {@apilink PlaywrightCrawlerOptions.requestQueue} constructor options, respectively.
*
- * If both {@link PlaywrightCrawlerOptions.requestList} and {@link PlaywrightCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * If both {@apilink PlaywrightCrawlerOptions.requestList} and {@apilink PlaywrightCrawlerOptions.requestQueue} are used,
+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
+ * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
- * The crawler finishes when there are no more {@link Request} objects to crawl.
+ * The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
- * `PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
- * and then calls the function provided by user as the {@link PlaywrightCrawlerOptions.requestHandler} option.
+ * `PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each {@apilink Request} object to crawl
+ * and then calls the function provided by user as the {@apilink PlaywrightCrawlerOptions.requestHandler} option.
*
* New pages are only opened when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the {@link PlaywrightCrawlerOptions.autoscaledPoolOptions}
+ * using the functionality provided by the {@apilink AutoscaledPool} class.
+ * All {@apilink AutoscaledPool} configuration options can be passed to the {@apilink PlaywrightCrawlerOptions.autoscaledPoolOptions}
* parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
- * {@link AutoscaledPoolOptions} are available directly in the `PlaywrightCrawler` constructor.
+ * {@apilink AutoscaledPoolOptions} are available directly in the `PlaywrightCrawler` constructor.
*
* Note that the pool of Playwright instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.
*
@@ -148,7 +148,6 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
*
* ```javascript
* const crawler = new PlaywrightCrawler({
- * requestList,
* async requestHandler({ page, request }) {
* // This function is called to extract data from a single web page
* // 'page' is an instance of Playwright.Page with page.goto(request.url) already called
@@ -169,7 +168,10 @@ export interface PlaywrightCrawlerOptions extends BrowserCrawlerOptions<
* },
* });
*
- * await crawler.run();
+ * await crawler.run([
+ * 'http://www.example.com/page-1',
+ * 'http://www.example.com/page-2',
+ * ]);
* ```
* @category Crawlers
*/
@@ -218,9 +220,9 @@ export class PlaywrightCrawler extends BrowserCrawler<{ browserPlugins: [Playwri
}
/**
- * Creates new {@link Router} instance that works based on request labels.
- * This instance can then serve as a `requestHandler` of your {@link PlaywrightCrawler}.
- * Defaults to the {@link PlaywrightCrawlingContext}.
+ * Creates new {@apilink Router} instance that works based on request labels.
+ * This instance can then serve as a `requestHandler` of your {@apilink PlaywrightCrawler}.
+ * Defaults to the {@apilink PlaywrightCrawlingContext}.
*
* > Serves as a shortcut for using `Router.create<PlaywrightCrawlingContext>()`.
*
diff --git a/packages/playwright-crawler/src/internals/playwright-launcher.ts b/packages/playwright-crawler/src/internals/playwright-launcher.ts
index 9dabd57d7..653e983eb 100644
--- a/packages/playwright-crawler/src/internals/playwright-launcher.ts
+++ b/packages/playwright-crawler/src/internals/playwright-launcher.ts
@@ -165,7 +165,7 @@ function getDefaultExecutablePath(launchContext: PlaywrightLaunchContext, config
* @param [launchContext]
* Optional settings passed to `browserType.launch()`. In addition to
* [Playwright's options](https://playwright.dev/docs/api/class-browsertype?_highlight=launch#browsertypelaunchoptions)
- * the object may contain our own {@link PlaywrightLaunchContext} that enable additional features.
+ * the object may contain our own {@apilink PlaywrightLaunchContext} that enable additional features.
* @param [config]
* @returns
* Promise that resolves to Playwright's `Browser` instance.
diff --git a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
index 5497815b6..6affaeb0f 100644
--- a/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
+++ b/packages/playwright-crawler/src/internals/utils/playwright-utils.ts
@@ -271,7 +271,7 @@ export async function blockRequests(page: Page, options: BlockRequestsOptions =
}
/**
- * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with {@link CheerioCrawler}.
+ * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with {@apilink CheerioCrawler}.
*
* **Example usage:**
* ```javascript
diff --git a/packages/puppeteer-crawler/README.md b/packages/puppeteer-crawler/README.md
new file mode 100644
index 000000000..6a5a3ff66
--- /dev/null
+++ b/packages/puppeteer-crawler/README.md
@@ -0,0 +1,54 @@
+# `@crawlee/puppeteer`
+
+Provides a simple framework for parallel crawling of web pages using headless Chrome with [Puppeteer](https://github.com/puppeteer/puppeteer). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
+
+Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.
+
+The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) or [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) constructor options, respectively.
+
+If both [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) and [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) are used,
+the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them
+to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.
+
+The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.
+
+`PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl
+and then calls the function provided by user as the [PuppeteerCrawlerOptions.requestHandler](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestHandler) option.
+
+New pages are only opened when there is enough free CPU and memory available,
+using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class.
+All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [PuppeteerCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#autoscaledPoolOptions)
+parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+[AutoscaledPoolOptions](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions) are available directly in the `PuppeteerCrawler` constructor.
+
+Note that the pool of Puppeteer instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.
+
+## Example usage
+
+```javascript
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ page, request }) {
+ // This function is called to extract data from a single web page
+ // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
+ // 'request' is an instance of Request class with information about the page to load
+ await Dataset.pushData({
+ title: await page.title(),
+ url: request.url,
+ succeeded: true,
+ })
+ },
+ async failedRequestHandler({ request }) {
+ // This function is called when the crawling of a request failed too many times
+ await Dataset.pushData({
+ url: request.url,
+ succeeded: false,
+ errors: request.errorMessages,
+ })
+ },
+});
+
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+]);
+```
diff --git a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
index 441e08808..739650542 100644
--- a/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
+++ b/packages/puppeteer-crawler/src/internals/enqueue-links/click-elements.ts
@@ -47,15 +47,15 @@ export interface EnqueueLinksByClickingElementsOptions {
requestQueue: RequestQueue;
/**
- * A CSS selector matching elements to be clicked on. Unlike in {@link enqueueLinks}, there is no default
+ * A CSS selector matching elements to be clicked on. Unlike in {@apilink enqueueLinks}, there is no default
* value. This is to prevent suboptimal use of this function by using it too broadly.
*/
selector: string;
- /** Sets {@link Request.userData} for newly enqueued requests. */
+ /** Sets {@apilink Request.userData} for newly enqueued requests. */
userData?: Dictionary;
- /** Sets {@link Request.label} for newly enqueued requests. */
+ /** Sets {@apilink Request.label} for newly enqueued requests. */
label?: string;
/**
@@ -68,7 +68,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* containing glob pattern strings matching the URLs to be enqueued.
*
* The plain objects must include at least the `glob` property, which holds the glob pattern string.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* The matching is always case-insensitive.
* If you need case-sensitive matching, use `regexps` property directly.
@@ -84,7 +84,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* containing regular expressions matching the URLs to be enqueued.
*
* The plain objects must include at least the `regexp` property, which holds the regular expression.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* If `regexps` is an empty array or `undefined`, then the function
* enqueues all the intercepted navigation requests produced by the page
@@ -96,11 +96,11 @@ export interface EnqueueLinksByClickingElementsOptions {
* *NOTE:* In future versions of SDK the options will be removed.
* Please use `globs` or `regexps` instead.
*
- * An array of {@link PseudoUrl} strings or plain objects
- * containing {@link PseudoUrl} strings matching the URLs to be enqueued.
+ * An array of {@apilink PseudoUrl} strings or plain objects
+ * containing {@apilink PseudoUrl} strings matching the URLs to be enqueued.
*
* The plain objects must include at least the `purl` property, which holds the pseudo-URL pattern string.
- * All remaining keys will be used as request options for the corresponding enqueued {@link Request} objects.
+ * All remaining keys will be used as request options for the corresponding enqueued {@apilink Request} objects.
*
* With a pseudo-URL string, the matching is always case-insensitive.
* If you need case-sensitive matching, use `regexps` property directly.
@@ -114,7 +114,7 @@ export interface EnqueueLinksByClickingElementsOptions {
pseudoUrls?: PseudoUrlInput[];
/**
- * Just before a new {@link Request} is constructed and enqueued to the {@link RequestQueue}, this function can be used
+ * Just before a new {@apilink Request} is constructed and enqueued to the {@apilink RequestQueue}, this function can be used
* to remove it or modify its contents such as `userData`, `payload` or, most importantly `uniqueKey`. This is useful
* when you need to enqueue multiple `Requests` to the queue that share the same URL, but differ in methods or payloads,
* or to dynamically update or create `userData`.
@@ -167,12 +167,12 @@ export interface EnqueueLinksByClickingElementsOptions {
* clicks all those elements using a mouse move and a left mouse button click and intercepts
* all the navigation requests that are subsequently produced by the page. The intercepted
* requests, including their methods, headers and payloads are then enqueued to a provided
- * {@link RequestQueue}. This is useful to crawl JavaScript heavy pages where links are not available
+ * {@apilink RequestQueue}. This is useful to crawl JavaScript heavy pages where links are not available
* in `href` elements, but rather navigations are triggered in click handlers.
- * If you're looking to find URLs in `href` attributes of the page, see {@link enqueueLinks}.
+ * If you're looking to find URLs in `href` attributes of the page, see {@apilink enqueueLinks}.
*
- * Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects
- * and override settings of the enqueued {@link Request} objects.
+ * Optionally, the function allows you to filter the target links' URLs using an array of {@apilink PseudoUrl} objects
+ * and override settings of the enqueued {@apilink Request} objects.
*
* **IMPORTANT**: To be able to do this, this function uses various mutations on the page,
* such as changing the Z-index of elements being clicked and their visibility. Therefore,
@@ -201,7 +201,7 @@ export interface EnqueueLinksByClickingElementsOptions {
* });
* ```
*
- * @returns Promise that resolves to {@link BatchAddRequestsResult} object.
+ * @returns Promise that resolves to {@apilink BatchAddRequestsResult} object.
*/
export async function enqueueLinksByClickingElements(options: EnqueueLinksByClickingElementsOptions): Promise<BatchAddRequestsResult> {
ow(options, ow.object.exactShape({
diff --git a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
index bbc07cd71..e66a3d710 100644
--- a/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
+++ b/packages/puppeteer-crawler/src/internals/puppeteer-crawler.ts
@@ -25,7 +25,7 @@ export interface PuppeteerCrawlerOptions extends BrowserCrawlerOptions<
{ browserPlugins: [PuppeteerPlugin] }
> {
/**
- * Options used by {@link launchPuppeteer} to start new Puppeteer instances.
+ * Options used by {@apilink launchPuppeteer} to start new Puppeteer instances.
*/
launchContext?: PuppeteerLaunchContext;
@@ -71,27 +71,27 @@ export interface PuppeteerCrawlerOptions extends BrowserCrawlerOptions<
*
* Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,
* it is useful for crawling of websites that require to execute JavaScript.
- * If the target website doesn't need JavaScript, consider using {@link CheerioCrawler},
+ * If the target website doesn't need JavaScript, consider using {@apilink CheerioCrawler},
* which downloads the pages using raw HTTP requests and is about 10x faster.
*
- * The source URLs are represented using {@link Request} objects that are fed from
- * {@link RequestList} or {@link RequestQueue} instances provided by the {@link PuppeteerCrawlerOptions.requestList}
- * or {@link PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.
+ * The source URLs are represented using {@apilink Request} objects that are fed from
+ * {@apilink RequestList} or {@apilink RequestQueue} instances provided by the {@apilink PuppeteerCrawlerOptions.requestList}
+ * or {@apilink PuppeteerCrawlerOptions.requestQueue} constructor options, respectively.
*
- * If both {@link PuppeteerCrawlerOptions.requestList} and {@link PuppeteerCrawlerOptions.requestQueue} are used,
- * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
- * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ * If both {@apilink PuppeteerCrawlerOptions.requestList} and {@apilink PuppeteerCrawlerOptions.requestQueue} are used,
+ * the instance first processes URLs from the {@apilink RequestList} and automatically enqueues all of them
+ * to {@apilink RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
*
- * The crawler finishes when there are no more {@link Request} objects to crawl.
+ * The crawler finishes when there are no more {@apilink Request} objects to crawl.
*
- * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@link Request} object to crawl
- * and then calls the function provided by user as the {@link PuppeteerCrawlerOptions.requestHandler} option.
+ * `PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each {@apilink Request} object to crawl
+ * and then calls the function provided by user as the {@apilink PuppeteerCrawlerOptions.requestHandler} option.
*
* New pages are only opened when there is enough free CPU and memory available,
- * using the functionality provided by the {@link AutoscaledPool} class.
- * All {@link AutoscaledPool} configuration options can be passed to the {@link PuppeteerCrawlerOptions.autoscaledPoolOptions}
+ * using the functionality provided by the {@apilink AutoscaledPool} class.
+ * All {@apilink AutoscaledPool} configuration options can be passed to the {@apilink PuppeteerCrawlerOptions.autoscaledPoolOptions}
* parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
- * {@link AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.
+ * {@apilink AutoscaledPoolOptions} are available directly in the `PuppeteerCrawler` constructor.
*
* Note that the pool of Puppeteer instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.
*
@@ -99,7 +99,6 @@ export interface PuppeteerCrawlerOptions extends BrowserCrawlerOptions<
*
* ```javascript
* const crawler = new PuppeteerCrawler({
- * requestList,
* async requestHandler({ page, request }) {
* // This function is called to extract data from a single web page
* // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
@@ -120,7 +119,10 @@ export interface PuppeteerCrawlerOptions extends BrowserCrawlerOptions<
* },
* });
*
- * await crawler.run();
+ * await crawler.run([
+ * 'http://www.example.com/page-1',
+ * 'http://www.example.com/page-2',
+ * ]);
* ```
* @category Crawlers
*/
@@ -169,9 +171,9 @@ export class PuppeteerCrawler extends BrowserCrawler<{ browserPlugins: [Puppetee
}
/**
- * Creates new {@link Router} instance that works based on request labels.
- * This instance can then serve as a `requestHandler` of your {@link PuppeteerCrawler}.
- * Defaults to the {@link PuppeteerCrawlingContext}.
+ * Creates new {@apilink Router} instance that works based on request labels.
+ * This instance can then serve as a `requestHandler` of your {@apilink PuppeteerCrawler}.
+ * Defaults to the {@apilink PuppeteerCrawlingContext}.
*
* > Serves as a shortcut for using `Router.create<PuppeteerCrawlingContext>()`.
*
diff --git a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
index 49505401e..3584f1d83 100644
--- a/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
+++ b/packages/puppeteer-crawler/src/internals/utils/puppeteer_utils.ts
@@ -165,7 +165,7 @@ export function injectJQuery(page: Page): Promise<unknown> {
}
/**
- * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with {@link CheerioCrawler}.
+ * Returns Cheerio handle for `page.content()`, allowing to work with the data same way as with {@apilink CheerioCrawler}.
*
* **Example usage:**
* ```javascript
@@ -332,7 +332,7 @@ export async function cacheResponses(page: Page, cache: Dictionary<Partial<Respo
* }
* ```
* Where `page` is a Puppeteer [`Page`](https://pptr.dev/api/puppeteer.page)
- * and `request` is a {@link Request}.
+ * and `request` is a {@apilink Request}.
*
* The function is compiled by using the `scriptString` parameter as the function's body,
* so any limitations to function bodies apply. Return value of the compiled function
diff --git a/packages/types/src/storages.ts b/packages/types/src/storages.ts
index 67e505115..7f8772458 100644
--- a/packages/types/src/storages.ts
+++ b/packages/types/src/storages.ts
@@ -2,7 +2,7 @@ import type { AllowedHttpMethods, Dictionary } from './utility-types';
/**
* A helper class that is used to report results from various
- * {@link RequestQueue} functions as well as {@link enqueueLinks}.
+ * {@apilink RequestQueue} functions as well as {@apilink enqueueLinks}.
*/
export interface QueueOperationInfo {
diff --git a/packages/utils/src/internals/memory-info.ts b/packages/utils/src/internals/memory-info.ts
index 37584025d..b4d9aa4ee 100644
--- a/packages/utils/src/internals/memory-info.ts
+++ b/packages/utils/src/internals/memory-info.ts
@@ -40,7 +40,7 @@ export interface MemoryInfo {
}
/**
- * Returns memory statistics of the process and the system, see {@link MemoryInfo}.
+ * Returns memory statistics of the process and the system, see {@apilink MemoryInfo}.
*
* If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits,
* otherwise it gets system memory limits.
diff --git a/packages/utils/src/internals/social.ts b/packages/utils/src/internals/social.ts
index 32c4c4f98..c4989ed72 100644
--- a/packages/utils/src/internals/social.ts
+++ b/packages/utils/src/internals/social.ts
@@ -215,7 +215,7 @@ export interface SocialHandles {
* The function removes duplicates from the resulting arrays and sorts the items alphabetically.
*
* Note that the `phones` field contains phone numbers extracted from the special phone links
- * such as `[call us](tel:+1234556789)` (see {@link phonesFromUrls})
+ * such as `[call us](tel:+1234556789)` (see {@apilink phonesFromUrls})
* and potentially other sources with high certainty, while `phonesUncertain` contains phone numbers
* extracted from the plain text, which might be very inaccurate.
*
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 1c48546b6..0a6015563 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -83,6 +83,7 @@ module.exports = {
{
projectRoot: `${__dirname}/..`,
changelogs: true,
+ readmes: true,
sortPackages: (a, b) => {
return packagesOrder.indexOf(a.packageName) - packagesOrder.indexOf(b.packageName);
},
@@ -112,7 +113,7 @@ module.exports = {
},
],
[
- require.resolve('docusaurus-gtm-plugin'),
+ 'docusaurus-gtm-plugin',
{
id: 'GTM-TKBX678',
},