Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Docs: fix various links (#625)

František Nesveda1cc63cd33043
diff --git a/README.md b/README.md
index 362066769..fa2e736e0 100644
--- a/README.md
+++ b/README.md
@@ -18,7 +18,7 @@ running on the [Apify Cloud](https://apify.com/).
## Motivation
-Thanks to tools like [Puppeteer](https://github.com/GoogleChrome/puppeteer) or
+Thanks to tools like [Puppeteer](https://github.com/puppeteer/puppeteer) or
[Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. But
eventually things will get complicated. For example, when you try to:
@@ -49,7 +49,7 @@ number of web pages using the [cheerio](https://www.npmjs.com/package/cheerio) H
efficient web crawler, but it does not work on websites that require JavaScript.
- [`PuppeteerCrawler`](https://sdk.apify.com/docs/api/puppeteer-crawler) - Enables the parallel crawling of
-a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/GoogleChrome/puppeteer).
+a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/puppeteer/puppeteer).
The pool of Chrome browsers is automatically scaled up and down based on available system resources.
- [`PuppeteerPool`](https://sdk.apify.com/docs/api/puppeteer-pool) - Provides web browser tabs for user jobs
diff --git a/docs/guides/data_storage.md b/docs/guides/data_storage.md
index bd1ccb192..05058dcfc 100644
--- a/docs/guides/data_storage.md
+++ b/docs/guides/data_storage.md
@@ -28,8 +28,8 @@ and output is stored in the default key-value store under the `INPUT` and `OUTPU
although it can be any other format.
In the Apify SDK, the key-value store is represented by the [`KeyValueStore`](/docs/api/key-value-store) class. In order to simplify access to the default
-key-value store, the SDK also provides [`Apify.getValue()`](/docs/api/apify#getValue) and
-[`Apify.setValue()`](/docs/api/apify#setValue) functions.
+key-value store, the SDK also provides [`Apify.getValue()`](/docs/api/apify#getvalue) and
+[`Apify.setValue()`](/docs/api/apify#setvalue) functions.
In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:
@@ -83,7 +83,7 @@ Each actor run is associated with a **default dataset**, which is created exclus
results specific for the actor run. Its usage is optional.
In the Apify SDK, the dataset is represented by the [`Dataset`](/docs/api/dataset) class. In order to simplify writes to the default dataset, the SDK
-also provides the [`Apify.pushData()`](/docs/api/apify#pushData) function.
+also provides the [`Apify.pushData()`](/docs/api/apify#pushdata) function.
In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:
diff --git a/docs/guides/environment_variables.md b/docs/guides/environment_variables.md
index d3b1f9bb0..32ef3261c 100644
--- a/docs/guides/environment_variables.md
+++ b/docs/guides/environment_variables.md
@@ -8,7 +8,7 @@ The following is a list of the environment variables used by Apify SDK that are
## `APIFY_HEADLESS`
If set to `1`, web browsers launched by Apify SDK will run in the headless mode. You can still override
this setting in the code, e.g. by passing the `headless: true` option to the
-[`Apify.launchPuppeteer()`](/docs/api/apify#launchPuppeteer) function. But having this setting
+[`Apify.launchPuppeteer()`](/docs/api/apify#launchpuppeteer) function. But having this setting
in an environment variable allows you to develop the crawler locally in headful mode to simplify the debugging,
and only run the crawler in headless mode once you deploy it to the Apify Cloud. By default, the browsers
are launched in headful mode, i.e. with windows.
@@ -16,7 +16,7 @@ are launched in headful mode, i.e. with windows.
## `APIFY_LOCAL_STORAGE_DIR`
Defines the path to a local directory where [`KeyValueStore`](/docs/api/key-value-store),
[`Dataset`](/docs/api/dataset), and [`RequestQueue`](/docs/api/request-queue) store their data.
-Typically it is set to `./apify_storage`. If omitted, you should define the [`APIFY_TOKEN`](#APIFY_TOKEN)
+Typically it is set to `./apify_storage`. If omitted, you should define the [`APIFY_TOKEN`](#apify_token)
environment variable instead.
## `APIFY_LOG_LEVEL`
@@ -39,5 +39,5 @@ in the Apify app. This feature is optional. You can use your own proxies or no p
## `APIFY_TOKEN`
The API token for your Apify Account. It is used to access the Apify API, e.g. to access cloud storage
or to run an actor in the Apify Cloud. You can find your API token on the
-[Account - Integrations](https://my.apify.com/account#integrations) page. If omitted,
+[Account - Integrations](https://my.apify.com/account#/integrations) page. If omitted,
you should define the `APIFY_LOCAL_STORAGE_DIR` environment variable instead.
diff --git a/docs/guides/getting_started.md b/docs/guides/getting_started.md
index 6e0fa15fc..4c89c1a92 100644
--- a/docs/guides/getting_started.md
+++ b/docs/guides/getting_started.md
@@ -205,7 +205,7 @@ Apify.main(async () => {
> If you're not familiar with the `async` and `await` keywords used in the example, trust that it is a native syntax in modern JavaScript and you can
> [learn more about it here](https://nikgrozev.com/2017/10/01/async-await/).
-The [`requestQueue.addRequest()`](/docs/api/request-queue#addRequest) function automatically converts the plain object we passed to it to a
+The [`requestQueue.addRequest()`](/docs/api/request-queue#addrequest) function automatically converts the plain object we passed to it to a
`Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://apify.com`. Now we need the
`handlePageFunction`.
@@ -420,7 +420,7 @@ const sameDomainLinks = absoluteUrls.filter(url => url.href.startsWith(ourDomain
#### Enqueueing links to `RequestQueue`
-This should be easy, because we already did that [earlier](#puttingitalltogether), remember? Just call `requestQueue.addRequest()` for all the new
+This should be easy, because we already did that [earlier](#putting-it-all-together), remember? Just call `requestQueue.addRequest()` for all the new
links. This will add them to the end of the queue for processing.
```js
@@ -516,8 +516,8 @@ No matter if you followed along with our coding or just copy pasted the resultin
should see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the
first enqueued page and so on and so on.
-> If you need help with running the code, refer back to the chapters on environment setup: [Setting up locally](#settinguplocally) and
-> [Setting up on the Apify Platform](#settingupontheapifyplatform).
+> If you need help with running the code, refer back to the chapters on environment setup: [Setting up locally](#setting-up-locally) and
+> [Setting up on the Apify Platform](#setting-up-on-the-apify-platform).
## Using Apify SDK to enqueue links like a boss
@@ -543,7 +543,7 @@ It also allows you to modify the resulting `Requests` to match your crawling nee
`enqueueLinks` is quite a powerful function so, like crawlers, it gets its arguments from an options object. This is useful, because you don't have to
remember their order! But also because we can easily extend its API and add new features. You can
-[find the full reference here](/docs/api/utils#enqueueLinks).
+[find the full reference here](/docs/api/utils#enqueuelinks).
We suggest using ES6 destructuring to grab the `enqueueLinks()` function off of the `utils` object, so you don't have to type `Apify.utils` all the
time.
@@ -1479,7 +1479,7 @@ Apify.main(async () => {
#### What's `Apify.pushData()`
-[`Apify.pushData()`](/docs/api/apify#pushData) is a helper function that saves data to the default [`Dataset`](/docs/api/dataset). `Dataset` is a
+[`Apify.pushData()`](/docs/api/apify#pushdata) is a helper function that saves data to the default [`Dataset`](/docs/api/dataset). `Dataset` is a
storage designed to hold virtually unlimited amount of data in a format similar to a table. Each time you call `Apify.pushData()` a new row in the
table is created, with the property names serving as column titles.
@@ -1534,7 +1534,7 @@ const input = await Apify.getInput();
```
On the Apify Platform, the actor's input that you can set in the Console is automatically saved to the default `KeyValueStore` under the key `INPUT`
-and by calling [`Apify.getInput()`](/docs/api/apify#getValue) you retrieve the value from the `KeyValueStore`.
+and by calling [`Apify.getInput()`](/docs/api/apify#getvalue) you retrieve the value from the `KeyValueStore`.
Running locally, you need to place an `INPUT.json` file in your default key value store for this to work.
diff --git a/docs/guides/motivation.md b/docs/guides/motivation.md
index 3d5932aab..1e50fd11b 100644
--- a/docs/guides/motivation.md
+++ b/docs/guides/motivation.md
@@ -3,7 +3,7 @@ id: motivation
title: Motivation
---
-Thanks to tools like [Puppeteer](https://github.com/GoogleChrome/puppeteer) or
+Thanks to tools like [Puppeteer](https://github.com/puppeteer/puppeteer) or
[Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. But
eventually things will get complicated. For example, when you try to:
diff --git a/docs/guides/puppeteer_live_view.md b/docs/guides/puppeteer_live_view.md
index 4847afcec..719821259 100644
--- a/docs/guides/puppeteer_live_view.md
+++ b/docs/guides/puppeteer_live_view.md
@@ -8,7 +8,7 @@ This is useful for debugging your crawlers that run in headless mode.
The live view dashboard is run on a web server that is started on a port specified by the `APIFY_CONTAINER_PORT` environment variable (typically
4321). To enable live view, pass the `useliveView: true` option to the `puppeteerPoolOptions` of
-[`PuppeteerCrawler`](/docs/api/puppeteer-crawler#new_PuppeteerCrawler_new):
+[`PuppeteerCrawler`](/docs/api/puppeteer-crawler#new-puppeteercrawleroptions):
```js
const crawler = new Apify.PuppeteerCrawler({
diff --git a/docs/readme/overview.md b/docs/readme/overview.md
index 79f7f0dbd..c1939a296 100644
--- a/docs/readme/overview.md
+++ b/docs/readme/overview.md
@@ -11,7 +11,7 @@ number of web pages using the [cheerio](https://www.npmjs.com/package/cheerio) H
efficient web crawler, but it does not work on websites that require JavaScript.
- [`PuppeteerCrawler`](https://sdk.apify.com/docs/api/puppeteer-crawler) - Enables the parallel crawling of
-a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/GoogleChrome/puppeteer).
+a large number of web pages using the headless Chrome browser and [Puppeteer](https://github.com/puppeteer/puppeteer).
The pool of Chrome browsers is automatically scaled up and down based on available system resources.
- [`PuppeteerPool`](https://sdk.apify.com/docs/api/puppeteer-pool) - Provides web browser tabs for user jobs
diff --git a/examples/call_actor.js b/examples/call_actor.js
index ca8489f9d..e638bfd52 100644
--- a/examples/call_actor.js
+++ b/examples/call_actor.js
@@ -7,7 +7,7 @@
*
* To make the example work, you'll need an [Apify Account](https://my.apify.com/).
* Go to [Account - Integrations](https://my.apify.com/account#/integrations) page to obtain your API token
- * and set it to the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable, or run the script using the CLI.
+ * and set it to the [`APIFY_TOKEN`](/docs/guides/environment-variables#apify_token) environment variable, or run the script using the CLI.
* If you deploy this actor to the Apify Cloud then you can set up a scheduler for early
* morning.
*
diff --git a/examples/puppeteer_with_proxy.js b/examples/puppeteer_with_proxy.js
index 362ba87ab..8c0cf706f 100644
--- a/examples/puppeteer_with_proxy.js
+++ b/examples/puppeteer_with_proxy.js
@@ -3,7 +3,7 @@
* over [Apify Proxy](https://docs.apify.com/proxy).
* To make it work, you'll need an Apify Account that has access to the proxy.
* The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app.
- * Just set it to the [`APIFY_PROXY_PASSWORD`](../guides/environment-variables#APIFY_PROXY_PASSWORD)
+ * Just set it to the [`APIFY_PROXY_PASSWORD`](../guides/environment-variables#apify_proxy_password)
* environment variable or run the script using the CLI.
*
* To run this example on the Apify Platform, select the `Node.js 12 + Chrome on Debian (apify/actor-node-chrome)` base image
diff --git a/examples/screenshots.js b/examples/screenshots.js
index 3caab512e..d26234b3d 100644
--- a/examples/screenshots.js
+++ b/examples/screenshots.js
@@ -1,6 +1,6 @@
/**
* This example demonstrates how to read and write data to the default key-value store using
- * [`Apify.getValue()`](/docs/api/apify#getValue) and [`Apify.setValue()`](/docs/api/apify#setValue).
+ * [`Apify.getValue()`](/docs/api/apify#getvalue) and [`Apify.setValue()`](/docs/api/apify#setvalue).
*
* The script crawls a list of URLs using Puppeteer,
* captures a screenshot of each page and saves it to the store. The list of URLs is
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 66750cc6e..84f0e5176 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -140,7 +140,7 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
/**
* Provides a simple framework for parallel crawling of web pages
- * using headless Chrome with [Puppeteer](https://github.com/GoogleChrome/puppeteer).
+ * using headless Chrome with [Puppeteer](https://github.com/puppeteer/puppeteer).
* The URLs to crawl are fed either from a static list of URLs
* or from a dynamic queue of URLs enabling recursive crawling of websites.
*
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 76b50e3cb..d199dc487 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -115,8 +115,8 @@ export const maybeStringify = (value, options) => {
* To access the input, you can also use the {@link Apify#getInput} convenience function.
*
* `KeyValueStore` stores its data either on local disk or in the Apify cloud,
- * depending on whether the [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR)
- * or [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variables are set.
+ * depending on whether the [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#apify_local_storage_dir)
+ * or [`APIFY_TOKEN`](/docs/guides/environment-variables#apify_token) environment variables are set.
*
* If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in
* the local directory in the following files:
@@ -127,12 +127,12 @@ export const maybeStringify = (value, options) => {
* unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable.
* The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.
*
- * If the [`APIFY_TOKEN`](/docs/guides/environment-variables#APIFY_TOKEN) environment variable is set but
- * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) not,
+ * If the [`APIFY_TOKEN`](/docs/guides/environment-variables#apify_token) environment variable is set but
+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#apify_local_storage_dir) not,
* the data is stored in the [Apify Key-value store](https://docs.apify.com/storage/key-value-store)
* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`
* option to {@link Apify#openKeyValueStore} function, even if the
- * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#APIFY_LOCAL_STORAGE_DIR) variable is set.
+ * [`APIFY_LOCAL_STORAGE_DIR`](/docs/guides/environment-variables#apify_local_storage_dir) variable is set.
*
* **Example usage:**
*
@@ -693,7 +693,7 @@ export const setValue = async (key, value, options) => {
/**
* Gets the actor input value from the default {@link KeyValueStore} associated with the current actor run.
*
- * This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](keyvaluestore#getvalue).
+ * This is just a convenient shortcut for [`keyValueStore.getValue('INPUT')`](key-value-store#getvalue).
* For example, calling the following code:
* ```javascript
* const input = await Apify.getInput();
diff --git a/src/puppeteer.js b/src/puppeteer.js
index 4b44feb69..ff3f2288d 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -159,7 +159,7 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
/**
* Launches headless Chrome using Puppeteer pre-configured to work within the Apify platform.
* The function has the same argument and the return value as `puppeteer.launch()`.
- * See <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions" target="_blank">
+ * See <a href="https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions" target="_blank">
* Puppeteer documentation</a> for more details.
*
* The `launchPuppeteer()` function alters the following Puppeteer options:
@@ -204,8 +204,8 @@ const getPuppeteerOrThrow = (puppeteerModule = 'puppeteer') => {
* [Apify Actor documentation](https://docs.apify.com/actor/build#base-images)
* for details.
*
- * For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)
- * or the [Puppeteer proxy Example](../examples/puppeteerwithproxy)
+ * For an example of usage, see the [Synchronous run Example](../examples/synchronous-run)
+ * or the [Puppeteer proxy Example](../examples/puppeteer-with-proxy)
*
* @param {LaunchPuppeteerOptions} [options]
* Optional settings passed to `puppeteer.launch()`. In addition to
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index b8d40ebbc..0f93b5d13 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -109,7 +109,7 @@ class PuppeteerInstance {
* @property {LaunchPuppeteerFunction} [launchPuppeteerFunction]
* Overrides the default function to launch a new Puppeteer instance.
* The function must return a promise resolving to
- * [`Browser`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-browser) instance.
+ * [`Browser`](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-browser) instance.
* See the source code on
* [GitHub](https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28)
* for the default implementation.
@@ -139,7 +139,7 @@ class PuppeteerInstance {
/**
* Manages a pool of Chrome browser instances controlled using
- * [Puppeteer](https://github.com/GoogleChrome/puppeteer).
+ * [Puppeteer](https://github.com/puppeteer/puppeteer).
*
* `PuppeteerPool` reuses Chrome instances and tabs using specific browser rotation and retirement policies.
* This is useful in order to facilitate rotation of proxies, cookies
diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.js
index 7f240afe8..79a296219 100644
--- a/src/puppeteer_utils.js
+++ b/src/puppeteer_utils.js
@@ -154,7 +154,7 @@ const injectJQuery = (page) => {
* });
* ```
*
- * @param {Page} page Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.
+ * @param {Page} page Puppeteer [Page](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-page) object.
* @return {Promise<*>}
* @memberOf puppeteer
*/
@@ -563,7 +563,7 @@ let logEnqueueLinksDeprecationWarning = true;
/**
* A namespace that contains various utilities for
- * [Puppeteer](https://github.com/GoogleChrome/puppeteer) - the headless Chrome Node API.
+ * [Puppeteer](https://github.com/puppeteer/puppeteer) - the headless Chrome Node API.
*
* **Example usage:**
*
diff --git a/website/pages/en/index.js b/website/pages/en/index.js
index 19fe9c67e..5c4da1722 100755
--- a/website/pages/en/index.js
+++ b/website/pages/en/index.js
@@ -166,10 +166,10 @@ const EasyCrawling = () => (
{[
{
content: 'There are three main classes that you can use to start crawling the web in no time. ' +
- 'Need to crawl plain HTML? Use the **blazing fast** [`CheerioCrawler`](docs/examples/cheeriocrawler).\n' +
+ 'Need to crawl plain HTML? Use the **blazing fast** [`CheerioCrawler`](docs/examples/cheerio-crawler).\n' +
'For complex websites that use **React**, **Vue** or other front-end javascript libraries and require JavaScript execution, ' +
- 'spawn a headless browser with [`PuppeteerCrawler`](docs/examples/puppeteercrawler).\n' +
- 'To **control all aspects** of your crawling, just use the bare bones [`BasicCrawler`](docs/examples/basiccrawler)',
+ 'spawn a headless browser with [`PuppeteerCrawler`](docs/examples/puppeteer-crawler).\n' +
+ 'To **control all aspects** of your crawling, just use the bare bones [`BasicCrawler`](docs/examples/basic-crawler)',
image: imgUrl('chrome_scrape.gif'),
imageAlign: 'right',
title: 'Easy crawling',
@@ -182,9 +182,9 @@ const PowerfulTools = () => (
<Block gridBlockAlign="left">
{[
{
- content: 'All the crawlers are automatically **scaled** based on available system resources using the [`AutoscaledPool`](docs/api/autoscaledpool) class. ' +
+ content: 'All the crawlers are automatically **scaled** based on available system resources using the [`AutoscaledPool`](docs/api/autoscaled-pool) class. ' +
'When you run your code on the [Apify Cloud](https://my.apify.com/actors), you can also take advantage of a [pool of proxies](https://apify.com/proxy) to avoid detection. ' +
- 'For data storage, you can use the [`Dataset`](docs/api/dataset), [`KeyValueStore`](docs/api/keyvaluestore) and [`RequestQueue`](docs/api/requestqueue) classes.',
+ 'For data storage, you can use the [`Dataset`](docs/api/dataset), [`KeyValueStore`](docs/api/key-value-store) and [`RequestQueue`](docs/api/request-queue) classes.',
image: imgUrl('source_code.png'),
imageAlign: 'left',
title: 'Powerful tools',
diff --git a/website/sidebars.json b/website/sidebars.json
index 7a86f6b91..bd08b961a 100644
--- a/website/sidebars.json
+++ b/website/sidebars.json
@@ -139,7 +139,7 @@
"label": "Return Types",
"ids": [
"typedefs/actor-run",
- "typedefs/apify-call-error",
+ "api/apify-call-error",
"typedefs/apify-env",
"typedefs/dataset-content",
"typedefs/memory-info",