Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fix broken links in docs

Ondra Urbanc9bc6a5f92e3
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index 435a8fba3..312e0e4e9 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -113,7 +113,8 @@ const DEFAULT_OPTIONS = {
* provided to the `options.handleFailedRequestFunction` parameter.
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
- * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
* Either `requestList` or `requestQueue` option must be provided (or both).
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 4aafe3609..c1dcdb4f1 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -132,7 +132,8 @@ const DEFAULT_OPTIONS = {
* provided to the `options.handleFailedRequestFunction` parameter.
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
- * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
* Either `requestList` or `requestQueue` option must be provided (or both).
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 2624e2ea9..ec5eae267 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -101,7 +101,8 @@ import { gotoExtended } from '../puppeteer_utils';
* provided to the `options.handleFailedRequestFunction` parameter.
* To make this work, you should **always**
* let your function throw exceptions rather than catch them.
- * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.
+ * The exceptions are logged to the request using the
+ * [`request.pushErrorMessage`](request#Request+pushErrorMessage) function.
* @param {RequestList} options.requestList
* Static list of URLs to be processed.
* Either `requestList` or `requestQueue` option must be provided (or both).
diff --git a/src/request.js b/src/request.js
index 31385c04f..88e1c4b15 100644
--- a/src/request.js
+++ b/src/request.js
@@ -248,6 +248,7 @@ class Request {
*
* @param {String} [message]
* @deprecated 2019/06/26
+ * @ignore
*/
doNotRetry(message) {
log.deprecated('request.doNotRetry is deprecated. Use request.noRetry = true; instead.');
diff --git a/src/request_queue.js b/src/request_queue.js
index d4f6ea47a..7bf9740b4 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -127,7 +127,7 @@ const getRequestId = (uniqueKey) => {
* @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.
* @property {Boolean} wasAlreadyHandled Indicates if request was already marked as handled.
* @property {String} requestId The ID of the added request
- * @property {Request|Object} request The original `Request` object passed to the `RequestQueue` function.
+ * @property {Object} request The original [`Request`](../api/request) object passed to the `RequestQueue` function.
*/
/**
@@ -319,14 +319,16 @@ export class RequestQueue {
/**
* Returns a next request in the queue to be processed, or `null` if there are no more pending requests.
*
- * Once you successfully finish processing of the request, you need to call {@link RequestQueue.markRequestHandled}
+ * Once you successfully finish processing of the request, you need to call
+ * [`requestQueue.markRequestHandled()`](#RequestQueue+markRequestHandled)
* to mark the request as handled in the queue. If there was some error in processing the request,
- * call {@link RequestQueue.reclaimRequest} instead, so that the queue will give the request to some other consumer
- * in another call to the `fetchNextRequest` function.
+ * call [`requestQueue.reclaimRequest()`](#RequestQueue+reclaimRequest) instead,
+ * so that the queue will give the request to some other consumer in another call to the `fetchNextRequest` function.
*
* Note that the `null` return value doesn't mean the queue processing finished,
* it means there are currently no pending requests.
- * To check whether all requests in queue were finished, use {@link RequestQueue.isFinished} instead.
+ * To check whether all requests in queue were finished,
+ * use [`requestQueue.isFinished()`](#RequestQueue+isFinished) instead.
*
* @returns {Promise<Request>}
* Returns the request object or `null` if there are no more pending requests.
@@ -389,8 +391,9 @@ export class RequestQueue {
}
/**
- * Marks a request that was previously returned by the {@link RequestQueue.fetchNextRequest} function
- * as handled after successful processing.
+ * Marks a request that was previously returned by the
+ * [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)
+ * function as handled after successful processing.
* Handled requests will never again be returned by the `fetchNextRequest` function.
*
* @param {Request} request
@@ -428,7 +431,7 @@ export class RequestQueue {
/**
* Reclaims a failed request back to the queue, so that it can be returned for processed later again
- * by another call to {@link RequestQueue.fetchNextRequest}.
+ * by another call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).
* The request record in the queue is updated using the provided `request` parameter.
* For example, this lets you store the number of retries or error messages for the request.
*
@@ -436,7 +439,8 @@ export class RequestQueue {
* @param {Object} [options]
* @param {Boolean} [options.forefront=false]
* If `true` then the request it placed to the beginning of the queue, so that it's returned
- * in the next call to {@link RequestQueue.fetchNextRequest}. By default, it's put to the end of the queue.
+ * in the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest).
+ * By default, it's put to the end of the queue.
* @return {Promise<QueueOperationInfo>}
*/
async reclaimRequest(request, options = {}) {
@@ -478,9 +482,10 @@ export class RequestQueue {
}
/**
- * Resolves to `true` if the next call to {@link RequestQueue.fetchNextRequest} would return `null`, otherwise it resolves to `false`.
+ * Resolves to `true` if the next call to [`requestQueue.fetchNextRequest()`](#RequestQueue+fetchNextRequest)
+ * would return `null`, otherwise it resolves to `false`.
* Note that even if the queue is empty, there might be some pending requests currently being processed.
- * If you need to ensure that there is no activity in the queue, use {@link RequestQueue.isFinished}.
+ * If you need to ensure that there is no activity in the queue, use [`requestQueue.isFinished()`](#RequestQueue+isFinished).
*
* @returns {Promise<Boolean>}
*/
diff --git a/src/utils_social.js b/src/utils_social.js
index db33d55b9..417c2b720 100644
--- a/src/utils_social.js
+++ b/src/utils_social.js
@@ -461,7 +461,7 @@ try {
* ```
*
* Note that the `phones` field contains phone numbers extracted from the special phone links
- * such as `<a href="tel:+1234556789">call us</a>` (see {@link phonesFromUrls})
+ * such as `<a href="tel:+1234556789">call us</a>` (see [`social.phonesFromUrls()`](#social.phonesFromUrls)])
* and potentially other sources with high certainty, while `phonesUncertain` contains phone numbers
* extracted from the plain text, which might be very inaccurate.
*
diff --git a/website/sidebars.json b/website/sidebars.json
index 9d55c0fb0..1ce196fb2 100644
--- a/website/sidebars.json
+++ b/website/sidebars.json
@@ -52,7 +52,6 @@
"typedefs/actorrun",
"typedefs/apifycallerror",
"typedefs/launchpuppeteeroptions",
- "typedefs/paginationlist",
"typedefs/queueoperationinfo"
]
}