Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: add v3.7 docs snapshot

Martin Adámek745f0d963497
diff --git a/website/versioned_docs/version-3.7/deployment/apify_platform.mdx b/website/versioned_docs/version-3.7/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.7/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.7/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.7/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.7/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.7/deployment/aws-browsers.md b/website/versioned_docs/version-3.7/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.7/deployment/aws-cheerio.md b/website/versioned_docs/version-3.7/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.7/deployment/gcp-browsers.md b/website/versioned_docs/version-3.7/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.7/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.7/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.7/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.7/examples/.eslintrc.json b/website/versioned_docs/version-3.7/examples/.eslintrc.json
new file mode 100644
index 000000000..d41f6dbc8
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/.eslintrc.json
@@ -0,0 +1,13 @@
+{
+ "root": true,
+ "extends": "@apify/ts",
+ "parserOptions": {
+ "project": "./tsconfig.eslint.json",
+ "ecmaVersion": 2022
+ },
+ "rules": {
+ "import/extensions": 0,
+ "import/no-extraneous-dependencies": 0,
+ "no-console": "off"
+ }
+}
diff --git a/website/versioned_docs/version-3.7/examples/accept_user_input.mdx b/website/versioned_docs/version-3.7/examples/accept_user_input.mdx
new file mode 100644
index 000000000..ddd872f0a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/accept_user_input.mdx
@@ -0,0 +1,23 @@
+---
+id: accept-user-input
+title: Accept user input
+---
+
+import CodeBlock from '@theme/CodeBlock';
+import AcceptInputSource from '!!raw-loader!./accept_user_input.ts';
+
+This example accepts and logs user input:
+
+<CodeBlock className="language-js">
+ {AcceptInputSource}
+</CodeBlock>
+
+To provide the actor with input, create a `INPUT.json` file inside the "default" key-value store:
+
+```bash
+{PROJECT_FOLDER}/storage/key_value_stores/default/INPUT.json
+```
+
+Anything in this file will be available to the actor when it runs.
+
+To learn about other ways to provide an actor with input, refer to the [Apify Platform Documentation](https://apify.com/docs/actor#run).
diff --git a/website/versioned_docs/version-3.7/examples/accept_user_input.ts b/website/versioned_docs/version-3.7/examples/accept_user_input.ts
new file mode 100644
index 000000000..25d4b48dd
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/accept_user_input.ts
@@ -0,0 +1,4 @@
+import { KeyValueStore } from 'crawlee';
+
+const input = await KeyValueStore.getInput();
+console.log(input);
diff --git a/website/versioned_docs/version-3.7/examples/add_data_to_dataset.mdx b/website/versioned_docs/version-3.7/examples/add_data_to_dataset.mdx
new file mode 100644
index 000000000..0bf29a3d8
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/add_data_to_dataset.mdx
@@ -0,0 +1,21 @@
+---
+id: add-data-to-dataset
+title: Add data to dataset
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import AddDataToDatasetSource from '!!raw-loader!roa-loader!./add_data_to_dataset.ts';
+
+This example saves data to the default dataset. If the dataset doesn't exist, it will be created.
+You can save data to custom datasets by using <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink>
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {AddDataToDatasetSource}
+</RunnableCodeBlock>
+
+Each item in this dataset will be saved to its own file in the following directory:
+
+```bash
+{PROJECT_FOLDER}/storage/datasets/default/
+```
diff --git a/website/versioned_docs/version-3.7/examples/add_data_to_dataset.ts b/website/versioned_docs/version-3.7/examples/add_data_to_dataset.ts
new file mode 100644
index 000000000..d57a0186e
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/add_data_to_dataset.ts
@@ -0,0 +1,21 @@
+import { Dataset, CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Function called for each URL
+ async requestHandler({ request, body }) {
+ // Save data to default dataset
+ await Dataset.pushData({
+ url: request.url,
+ html: body,
+ });
+ },
+});
+
+await crawler.addRequests([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+ 'http://www.example.com/page-3',
+]);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/basic_crawler.mdx b/website/versioned_docs/version-3.7/examples/basic_crawler.mdx
new file mode 100644
index 000000000..81a53ee9f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/basic_crawler.mdx
@@ -0,0 +1,19 @@
+---
+id: basic-crawler
+title: Basic crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import BasicCrawlerSource from '!!raw-loader!roa-loader!./basic_crawler.ts';
+
+This is the most bare-bones example of using Crawlee, which demonstrates some of its building blocks such as the <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>. You probably don't need to go this deep though, and it would be better to start with one of the full-featured crawlers
+like <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> or <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.
+
+The script simply downloads several web pages with plain HTTP requests using the <ApiLink to="basic-crawler/interface/BasicCrawlingContext#sendRequest">`sendRequest`</ApiLink> utility function (which uses the [`got-scraping`](https://github.com/apify/got-scraping)
+npm module internally) and stores their raw HTML and URL in the default dataset. In local configuration, the data will be stored as JSON files in
+`./storage/datasets/default`.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {BasicCrawlerSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/basic_crawler.ts b/website/versioned_docs/version-3.7/examples/basic_crawler.ts
new file mode 100644
index 000000000..f1e4584a4
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/basic_crawler.ts
@@ -0,0 +1,35 @@
+import { BasicCrawler, Dataset } from 'crawlee';
+
+// Create a BasicCrawler - the simplest crawler that enables
+// users to implement the crawling logic themselves.
+const crawler = new BasicCrawler({
+ // This function will be called for each URL to crawl.
+ async requestHandler({ request, sendRequest, log }) {
+ const { url } = request;
+ log.info(`Processing ${url}...`);
+
+ // Fetch the page HTML via the crawlee sendRequest utility method
+ // By default, the method will use the current request that is being handled, so you don't have to
+ // provide it yourself. You can also provide a custom request if you want.
+ const { body } = await sendRequest();
+
+ // Store the HTML and URL to the default dataset.
+ await Dataset.pushData({
+ url,
+ html: body,
+ });
+ },
+});
+
+// The initial list of URLs to crawl. Here we use just a few hard-coded URLs.
+await crawler.addRequests([
+ 'https://www.google.com',
+ 'https://www.example.com',
+ 'https://www.bing.com',
+ 'https://www.wikipedia.com',
+]);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
+
+console.log('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/cheerio_crawler.mdx b/website/versioned_docs/version-3.7/examples/cheerio_crawler.mdx
new file mode 100644
index 000000000..434488ae2
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/cheerio_crawler.mdx
@@ -0,0 +1,14 @@
+---
+id: cheerio-crawler
+title: Cheerio crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CheerioCrawlerSource from '!!raw-loader!roa-loader!./cheerio_crawler.ts';
+
+This example demonstrates how to use <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, parse the HTML using the [Cheerio library](https://www.npmjs.com/package/cheerio) and extract some data from it: the page title and all `h1` tags.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CheerioCrawlerSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts
new file mode 100644
index 000000000..c496c1572
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts
@@ -0,0 +1,64 @@
+import { Dataset, CheerioCrawler, log, LogLevel } from 'crawlee';
+
+// Crawlers come with various utilities, e.g. for logging.
+// Here we use debug level of logging to improve the debugging experience.
+// This functionality is optional!
+log.setLevel(LogLevel.DEBUG);
+
+// Create an instance of the CheerioCrawler class - a crawler
+// that automatically loads the URLs and parses their HTML using the cheerio library.
+const crawler = new CheerioCrawler({
+ // The crawler downloads and processes the web pages in parallel, with a concurrency
+ // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // Here we define some hard limits for the concurrency.
+ minConcurrency: 10,
+ maxConcurrency: 50,
+
+ // On error, retry each page at most once.
+ maxRequestRetries: 1,
+
+ // Increase the timeout for processing of each page.
+ requestHandlerTimeoutSecs: 30,
+
+ // Limit to 10 requests per one crawl
+ maxRequestsPerCrawl: 10,
+
+ // This function will be called for each URL to crawl.
+ // It accepts a single parameter, which is an object with options as:
+ // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler
+ // We use for demonstration only 2 of them:
+ // - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method
+ // - $: the cheerio object containing parsed HTML
+ async requestHandler({ request, $ }) {
+ log.debug(`Processing ${request.url}...`);
+
+ // Extract data from the page using cheerio.
+ const title = $('title').text();
+ const h1texts: { text: string }[] = [];
+ $('h1').each((index, el) => {
+ h1texts.push({
+ text: $(el).text(),
+ });
+ });
+
+ // Store the results to the dataset. In local configuration,
+ // the data will be stored as JSON files in ./storage/datasets/default
+ await Dataset.pushData({
+ url: request.url,
+ title,
+ h1texts,
+ });
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries + 1 times.
+ failedRequestHandler({ request }) {
+ log.debug(`Request ${request.url} failed twice.`);
+ },
+});
+
+// Run the crawler and wait for it to finish.
+await crawler.run([
+ 'https://crawlee.dev',
+]);
+
+log.debug('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/crawl_all_links.mdx b/website/versioned_docs/version-3.7/examples/crawl_all_links.mdx
new file mode 100644
index 000000000..85bf49471
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_all_links.mdx
@@ -0,0 +1,63 @@
+---
+id: crawl-all-links
+title: Crawl all links on a website
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+
+import CheerioSource from '!!raw-loader!roa-loader!./crawl_all_links_cheerio.ts';
+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteer.ts';
+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';
+
+This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`
+as the crawler navigates from page to page.
+
+:::tip
+
+If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the <ApiLink to="core/interface/EnqueueLinksOptions#strategy">`strategy`</ApiLink>
+option. You can find more info about this option in the [`Crawl relative links`](./crawl-relative-links) examples.
+
+:::
+
+<Tabs groupId="crawler-type">
+
+<TabItem value="cheerio_crawler" label="Cheerio Crawler" default>
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CheerioSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="puppeteer_crawler" label="Puppeteer Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="playwright_crawler" label="Playwright Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-playwright-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="playwright">
+ {PlaywrightSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/examples/crawl_all_links_cheerio.ts b/website/versioned_docs/version-3.7/examples/crawl_all_links_cheerio.ts
new file mode 100644
index 000000000..e3c930bc2
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_all_links_cheerio.ts
@@ -0,0 +1,13 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ // Add all links from page to RequestQueue
+ await enqueueLinks();
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_all_links_playwright.ts b/website/versioned_docs/version-3.7/examples/crawl_all_links_playwright.ts
new file mode 100644
index 000000000..bf0440dee
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_all_links_playwright.ts
@@ -0,0 +1,13 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ // Add all links from page to RequestQueue
+ await enqueueLinks();
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_all_links_puppeteer.ts b/website/versioned_docs/version-3.7/examples/crawl_all_links_puppeteer.ts
new file mode 100644
index 000000000..cf6fcb55c
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_all_links_puppeteer.ts
@@ -0,0 +1,13 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ // Add all links from page to RequestQueue
+ await enqueueLinks();
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_multiple_urls.mdx b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls.mdx
new file mode 100644
index 000000000..e6510cff9
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls.mdx
@@ -0,0 +1,54 @@
+---
+id: crawl-multiple-urls
+title: Crawl multiple URLs
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+
+import CheerioSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_cheerio.ts';
+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_puppeteer.ts';
+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_playwright.ts';
+
+This example crawls the specified list of URLs.
+
+<Tabs groupId="crawler-type">
+
+<TabItem value="cheerio_crawler" label="Cheerio Crawler" default>
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CheerioSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="puppeteer_crawler" label="Puppeteer Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="playwright_crawler" label="Playwright Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-playwright-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="playwright">
+ {PlaywrightSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_cheerio.ts b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_cheerio.ts
new file mode 100644
index 000000000..011d8b7d6
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_cheerio.ts
@@ -0,0 +1,16 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Function called for each URL
+ async requestHandler({ request, $, log }) {
+ const title = $('title').text();
+ log.info(`URL: ${request.url}\nTITLE: ${title}`);
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+ 'http://www.example.com/page-3',
+]);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_playwright.ts b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_playwright.ts
new file mode 100644
index 000000000..9581bea3f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_playwright.ts
@@ -0,0 +1,16 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ // Function called for each URL
+ async requestHandler({ request, page, log }) {
+ const title = await page.title();
+ log.info(`URL: ${request.url}\nTITLE: ${title}`);
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+ 'http://www.example.com/page-3',
+]);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_puppeteer.ts b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_puppeteer.ts
new file mode 100644
index 000000000..d7c6a99a5
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_multiple_urls_puppeteer.ts
@@ -0,0 +1,16 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ // Function called for each URL
+ async requestHandler({ request, page, log }) {
+ const title = await page.title();
+ log.info(`URL: ${request.url}\nTITLE: ${title}`);
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run([
+ 'http://www.example.com/page-1',
+ 'http://www.example.com/page-2',
+ 'http://www.example.com/page-3',
+]);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_relative_links.mdx b/website/versioned_docs/version-3.7/examples/crawl_relative_links.mdx
new file mode 100644
index 000000000..7281f0c7a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_relative_links.mdx
@@ -0,0 +1,88 @@
+---
+id: crawl-relative-links
+title: Crawl a website with relative links
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+
+import AllLinksSource from '!!raw-loader!roa-loader!./crawl_relative_links_all.ts';
+import SameDomainSource from '!!raw-loader!roa-loader!./crawl_relative_links_same_domain.ts';
+import SameHostnameSource from '!!raw-loader!roa-loader!./crawl_relative_links_same_hostname.ts';
+
+When crawling a website, you may encounter different types of links present that you may want to crawl.
+To facilitate the easy crawling of such links, we provide the `enqueueLinks()` method on the crawler context, which will
+automatically find links and add them to the crawler's <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>.
+
+We provide 3 different strategies for crawling relative links:
+
+- <ApiLink to="core/enum/EnqueueStrategy#All"><inlineCode>All</inlineCode> (or the string <inlineCode>"all"</inlineCode>)</ApiLink> which will
+enqueue all links found, regardless of the domain they point to.
+- <ApiLink to="core/enum/EnqueueStrategy#SameHostname"><inlineCode>SameHostname</inlineCode> (or the string <inlineCode>"same-hostname"</inlineCode>)</ApiLink> which
+will enqueue all links found for the same hostname. This is the default strategy.
+- <ApiLink to="core/enum/EnqueueStrategy#SameDomain"><inlineCode>SameDomain</inlineCode> (or the string <inlineCode>"same-domain"</inlineCode>)</ApiLink> which
+will enqueue all links found that have the same domain name, including links from any possible subdomain.
+
+:::note
+
+For these examples, we are using the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, however
+the same method is available for both the <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>
+and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, and you use it
+the exact same way.
+
+:::
+
+<Tabs groupId="enqueue_strategy">
+
+<TabItem value="all" label="All Links">
+
+:::note Example domains
+
+Any urls found will be matched by this strategy, even if they go off of the site you are currently crawling.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {AllLinksSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="same_hostname" label="Same Hostname">
+
+:::note Example domains
+
+For a url of `https://example.com`, `enqueueLinks()` will match relative urls and urls that point to the same
+hostname.
+
+> This is the default strategy when calling `enqueueLinks()`, so you don't have to specify it.
+
+For instance, hyperlinks like `https://example.com/some/path`, `/absolute/example` or `./relative/example` will all be matched by this strategy. But links to any subdomain like `https://subdomain.example.com/some/path` won't.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {SameHostnameSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="same-subdomain" label="Same Subdomain" default>
+
+:::note Example domains
+
+For a url of `https://subdomain.example.com`, `enqueueLinks()` will match relative urls or urls that point to the same domain name, regardless of their subdomain.
+
+For instance, hyperlinks like `https://subdomain.example.com/some/path`, `/absolute/example` or `./relative/example` will all be matched by this strategy, as well as links to other subdomains or to the naked domain, like `https://other-subdomain.example.com` or `https://example.com` will work too.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {SameDomainSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/examples/crawl_relative_links_all.ts b/website/versioned_docs/version-3.7/examples/crawl_relative_links_all.ts
new file mode 100644
index 000000000..e4401cb9f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_relative_links_all.ts
@@ -0,0 +1,18 @@
+import { CheerioCrawler, EnqueueStrategy } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ await enqueueLinks({
+ // Setting the strategy to 'all' will enqueue all links found
+ // highlight-next-line
+ strategy: EnqueueStrategy.All,
+ // Alternatively, you can pass in the string 'all'
+ // strategy: 'all',
+ });
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_domain.ts b/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_domain.ts
new file mode 100644
index 000000000..59c833c05
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_domain.ts
@@ -0,0 +1,19 @@
+import { CheerioCrawler, EnqueueStrategy } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ await enqueueLinks({
+ // Setting the strategy to 'same-domain' will enqueue all links found that are on the
+ // same hostname as request.loadedUrl or request.url
+ // highlight-next-line
+ strategy: EnqueueStrategy.SameDomain,
+ // Alternatively, you can pass in the string 'same-domain'
+ // strategy: 'same-domain',
+ });
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_hostname.ts b/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_hostname.ts
new file mode 100644
index 000000000..052f24dcf
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_relative_links_same_hostname.ts
@@ -0,0 +1,19 @@
+import { CheerioCrawler, EnqueueStrategy } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ await enqueueLinks({
+ // Setting the strategy to 'same-hostname' will enqueue all links found that are on the
+ // same hostname (including subdomain) as request.loadedUrl or request.url
+ // highlight-next-line
+ strategy: EnqueueStrategy.SameHostname,
+ // Alternatively, you can pass in the string 'same-hostname'
+ // strategy: 'same-hostname',
+ });
+ },
+});
+
+// Run the crawler with initial request
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_single_url.mdx b/website/versioned_docs/version-3.7/examples/crawl_single_url.mdx
new file mode 100644
index 000000000..d15e1d8f5
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_single_url.mdx
@@ -0,0 +1,17 @@
+---
+id: crawl-single-url
+title: Crawl a single URL
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./crawl_single_url.ts';
+
+This example uses the [`got-scraping`](https://github.com/apify/got-scraping) npm package
+to grab the HTML of a web page.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CrawlSource}
+</RunnableCodeBlock>
+
+If you don't want to hard-code the URL into the script, refer to the [Accept User Input](./accept-user-input) example.
diff --git a/website/versioned_docs/version-3.7/examples/crawl_single_url.ts b/website/versioned_docs/version-3.7/examples/crawl_single_url.ts
new file mode 100644
index 000000000..f588fd445
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_single_url.ts
@@ -0,0 +1,5 @@
+import { gotScraping } from 'got-scraping';
+
+// Get the HTML of a web page
+const { body } = await gotScraping({ url: 'https://www.example.com' });
+console.log(body);
diff --git a/website/versioned_docs/version-3.7/examples/crawl_sitemap.mdx b/website/versioned_docs/version-3.7/examples/crawl_sitemap.mdx
new file mode 100644
index 000000000..a136f00ee
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_sitemap.mdx
@@ -0,0 +1,55 @@
+---
+id: crawl-sitemap
+title: Crawl a sitemap
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+
+import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';
+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';
+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';
+
+This example downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.
+
+<Tabs groupId="crawler-type">
+
+<TabItem value="cheerio_crawler" label="Cheerio Crawler" default>
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CheerioSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="puppeteer_crawler" label="Puppeteer Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="playwright_crawler" label="Playwright Crawler">
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-playwright-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="playwright">
+ {PlaywrightSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/examples/crawl_sitemap_cheerio.ts b/website/versioned_docs/version-3.7/examples/crawl_sitemap_cheerio.ts
new file mode 100644
index 000000000..d59853af1
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_sitemap_cheerio.ts
@@ -0,0 +1,16 @@
+import { CheerioCrawler, Sitemap } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Function called for each URL
+ async requestHandler({ request, log }) {
+ log.info(request.url);
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl a sitemap)
+});
+
+const { urls } = await Sitemap.load('https://crawlee.dev/sitemap.xml');
+
+await crawler.addRequests(urls);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/crawl_sitemap_playwright.ts b/website/versioned_docs/version-3.7/examples/crawl_sitemap_playwright.ts
new file mode 100644
index 000000000..427399d4c
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_sitemap_playwright.ts
@@ -0,0 +1,16 @@
+import { PlaywrightCrawler, Sitemap } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ // Function called for each URL
+ async requestHandler({ request, log }) {
+ log.info(request.url);
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl a sitemap)
+});
+
+const { urls } = await Sitemap.load('https://crawlee.dev/sitemap.xml');
+
+await crawler.addRequests(urls);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/crawl_sitemap_puppeteer.ts b/website/versioned_docs/version-3.7/examples/crawl_sitemap_puppeteer.ts
new file mode 100644
index 000000000..7718bd02b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_sitemap_puppeteer.ts
@@ -0,0 +1,16 @@
+import { PuppeteerCrawler, Sitemap } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ // Function called for each URL
+ async requestHandler({ request, log }) {
+ log.info(request.url);
+ },
+ maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl a sitemap)
+});
+
+const { urls } = await Sitemap.load('https://crawlee.dev/sitemap.xml');
+
+await crawler.addRequests(urls);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/crawl_some_links.mdx b/website/versioned_docs/version-3.7/examples/crawl_some_links.mdx
new file mode 100644
index 000000000..fb9cde716
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_some_links.mdx
@@ -0,0 +1,14 @@
+---
+id: crawl-some-links
+title: Crawl some links on a website
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';
+
+This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="core/interface/EnqueueLinksOptions#globs">`globs`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/crawl_some_links.ts b/website/versioned_docs/version-3.7/examples/crawl_some_links.ts
new file mode 100644
index 000000000..51912bb10
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawl_some_links.ts
@@ -0,0 +1,21 @@
+import { CheerioCrawler } from 'crawlee';
+
+// Create a CheerioCrawler
+const crawler = new CheerioCrawler({
+ // Limits the crawler to only 10 requests (do not use if you want to crawl all links)
+ maxRequestsPerCrawl: 10,
+ // Function called for each URL
+ async requestHandler({ request, enqueueLinks, log }) {
+ log.info(request.url);
+ // Add some links from page to the crawler's RequestQueue
+ await enqueueLinks({
+ globs: ['http?(s)://crawlee.dev/*/*'],
+ });
+ },
+});
+
+// Define the starting URL
+await crawler.addRequests(['https://crawlee.dev']);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/crawler-plugins/index.mdx b/website/versioned_docs/version-3.7/examples/crawler-plugins/index.mdx
new file mode 100644
index 000000000..08bd4d943
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawler-plugins/index.mdx
@@ -0,0 +1,78 @@
+---
+id: playwright-puppeteer-extra
+title: Using puppeteer-extra and playwright-extra
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import ApiLink from '@site/src/components/ApiLink';
+import PuppeteerExtraSource from '!!raw-loader!roa-loader!./puppeteer-extra.ts';
+import PlaywrightExtraSource from '!!raw-loader!roa-loader!./playwright-extra.ts';
+
+[`puppeteer-extra`](https://www.npmjs.com/package/puppeteer-extra) and [`playwright-extra`](https://www.npmjs.com/package/playwright-extra) are community-built
+libraries that bring in a plugin system to enhance the usage of [`puppeteer`](https://www.npmjs.com/package/puppeteer) and
+[`playwright`](https://www.npmjs.com/package/playwright) respectively (bringing in extra functionality, like improving stealth for
+example by using the [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) plugin).
+
+:::tip Available plugins
+
+You can see a list of available plugins on the [`puppeteer-extra`s plugin list](https://www.npmjs.com/package/puppeteer-extra#plugins).
+
+For [`playwright`](https://www.npmjs.com/package/playwright), please see [`playwright-extra`s plugin list](https://www.npmjs.com/package/playwright-extra#plugins) instead.
+
+:::
+
+In this example, we'll show you how to use the [`puppeteer-extra-plugin-stealth`](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) plugin
+to help you avoid bot detections when crawling your target website.
+
+<Tabs>
+<TabItem value="puppeteer" label="Puppeteer & puppeteer-extra" default>
+
+:::info Before you begin
+
+Make sure you've installed the `puppeteer-extra` and `puppeteer-extra-plugin-stealth` packages via your preferred package manager
+
+```bash
+npm install puppeteer-extra puppeteer-extra-plugin-stealth
+```
+
+:::
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" title="src/crawler.ts" type="puppeteer">
+ {PuppeteerExtraSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="playwright" label="Playwright & playwright-extra">
+
+:::info Before you begin
+
+Make sure you've installed the `playwright-extra` and `puppeteer-extra-plugin-stealth` packages via your preferred package manager
+
+```bash
+npm install playwright-extra puppeteer-extra-plugin-stealth
+```
+
+:::
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" title="src/crawler.ts" type="playwright">
+ {PlaywrightExtraSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/examples/crawler-plugins/playwright-extra.ts b/website/versioned_docs/version-3.7/examples/crawler-plugins/playwright-extra.ts
new file mode 100644
index 000000000..c2b733f47
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawler-plugins/playwright-extra.ts
@@ -0,0 +1,74 @@
+import { Dataset, PlaywrightCrawler } from 'crawlee';
+
+// For playwright-extra you will need to import the browser type itself that you want to use!
+// By default, PlaywrightCrawler uses chromium, but you can also use firefox or webkit.
+import { chromium } from 'playwright-extra';
+import stealthPlugin from 'puppeteer-extra-plugin-stealth';
+
+// First, we tell playwright-extra to use the plugin (or plugins) we want.
+// Certain plugins might have options you can pass in - read up on their documentation!
+chromium.use(stealthPlugin());
+
+// Create an instance of the PlaywrightCrawler class - a crawler
+// that automatically loads the URLs in headless Chrome / Playwright.
+const crawler = new PlaywrightCrawler({
+ launchContext: {
+ // !!! You need to specify this option to tell Crawlee to use playwright-extra as the launcher !!!
+ launcher: chromium,
+ launchOptions: {
+ // Other playwright options work as usual
+ headless: true,
+ },
+ },
+
+ // Stop crawling after several pages
+ maxRequestsPerCrawl: 50,
+
+ // This function will be called for each URL to crawl.
+ // Here you can write the Puppeteer scripts you are familiar with,
+ // with the exception that browsers and pages are automatically managed by Crawlee.
+ // The function accepts a single parameter, which is an object with the following fields:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ log.info(`Processing ${request.url}...`);
+
+ // A function to be evaluated by Puppeteer within the browser context.
+ const data = await page.$$eval('.athing', ($posts) => {
+ const scrapedData: { title: string; rank: string; href: string }[] = [];
+
+ // We're getting the title, rank and URL of each post on Hacker News.
+ $posts.forEach(($post) => {
+ scrapedData.push({
+ title: $post.querySelector('.title a').innerText,
+ rank: $post.querySelector('.rank').innerText,
+ href: $post.querySelector('.title a').href,
+ });
+ });
+
+ return scrapedData;
+ });
+
+ // Store the results to the default dataset.
+ await Dataset.pushData(data);
+
+ // Find a link to the next page and enqueue it if it exists.
+ const infos = await enqueueLinks({
+ selector: '.morelink',
+ });
+
+ if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ failedRequestHandler({ request, log }) {
+ log.error(`Request ${request.url} failed too many times.`);
+ },
+});
+
+await crawler.addRequests(['https://news.ycombinator.com/']);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
+
+console.log('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/crawler-plugins/puppeteer-extra.ts b/website/versioned_docs/version-3.7/examples/crawler-plugins/puppeteer-extra.ts
new file mode 100644
index 000000000..6da7bf8a8
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/crawler-plugins/puppeteer-extra.ts
@@ -0,0 +1,71 @@
+import { Dataset, PuppeteerCrawler } from 'crawlee';
+import puppeteerExtra from 'puppeteer-extra';
+import stealthPlugin from 'puppeteer-extra-plugin-stealth';
+
+// First, we tell puppeteer-extra to use the plugin (or plugins) we want.
+// Certain plugins might have options you can pass in - read up on their documentation!
+puppeteerExtra.use(stealthPlugin());
+
+// Create an instance of the PuppeteerCrawler class - a crawler
+// that automatically loads the URLs in headless Chrome / Puppeteer.
+const crawler = new PuppeteerCrawler({
+ launchContext: {
+ // !!! You need to specify this option to tell Crawlee to use puppeteer-extra as the launcher !!!
+ launcher: puppeteerExtra,
+ launchOptions: {
+ // Other puppeteer options work as usual
+ headless: true,
+ },
+ },
+
+ // Stop crawling after several pages
+ maxRequestsPerCrawl: 50,
+
+ // This function will be called for each URL to crawl.
+ // Here you can write the Puppeteer scripts you are familiar with,
+ // with the exception that browsers and pages are automatically managed by Crawlee.
+ // The function accepts a single parameter, which is an object with the following fields:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ log.info(`Processing ${request.url}...`);
+
+ // A function to be evaluated by Puppeteer within the browser context.
+ const data = await page.$$eval('.athing', ($posts) => {
+ const scrapedData: { title: string; rank: string; href: string }[] = [];
+
+ // We're getting the title, rank and URL of each post on Hacker News.
+ $posts.forEach(($post) => {
+ scrapedData.push({
+ title: $post.querySelector('.title a').innerText,
+ rank: $post.querySelector('.rank').innerText,
+ href: $post.querySelector('.title a').href,
+ });
+ });
+
+ return scrapedData;
+ });
+
+ // Store the results to the default dataset.
+ await Dataset.pushData(data);
+
+ // Find a link to the next page and enqueue it if it exists.
+ const infos = await enqueueLinks({
+ selector: '.morelink',
+ });
+
+ if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ failedRequestHandler({ request, log }) {
+ log.error(`Request ${request.url} failed too many times.`);
+ },
+});
+
+await crawler.addRequests(['https://news.ycombinator.com/']);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
+
+console.log('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/export_entire_dataset.mdx b/website/versioned_docs/version-3.7/examples/export_entire_dataset.mdx
new file mode 100644
index 000000000..bea044b92
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/export_entire_dataset.mdx
@@ -0,0 +1,14 @@
+---
+id: export-entire-dataset
+title: Export entire dataset to one file
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./export_entire_dataset.ts';
+
+This `Dataset` example uses the `exportToValue` function to export the entire default dataset to a single CSV file into the default key-value store.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/export_entire_dataset.ts b/website/versioned_docs/version-3.7/examples/export_entire_dataset.ts
new file mode 100644
index 000000000..c912a9e5c
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/export_entire_dataset.ts
@@ -0,0 +1,20 @@
+import { Dataset } from 'crawlee';
+
+// Retrieve or generate two items to be pushed
+const data = [
+ {
+ id: 123,
+ name: 'foo',
+ },
+ {
+ id: 456,
+ name: 'bar',
+ },
+];
+
+// Push the two items to the default dataset
+await Dataset.pushData(data);
+
+// Export the entirety of the dataset to a single file in
+// the default key-value store under the key "OUTPUT"
+await Dataset.exportToCSV('OUTPUT');
diff --git a/website/versioned_docs/version-3.7/examples/forms.mdx b/website/versioned_docs/version-3.7/examples/forms.mdx
new file mode 100644
index 000000000..9d3250afa
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/forms.mdx
@@ -0,0 +1,24 @@
+---
+id: forms
+title: Forms
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./forms.ts';
+
+This example demonstrates how to use <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> to
+automatically fill and submit a search form to look up repositories on [GitHub](https://github.com) using headless Chrome / Puppeteer.
+The crawler first fills in the search term, repository owner, start date and language of the repository, then submits the form
+and prints out the results. Finally, the results are saved either on the Apify platform to the
+default <ApiLink to="core/class/Dataset">`dataset`</ApiLink> or on the local machine as JSON files in `./storage/datasets/default`.
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/forms.ts b/website/versioned_docs/version-3.7/examples/forms.ts
new file mode 100644
index 000000000..29677f91d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/forms.ts
@@ -0,0 +1,39 @@
+import { Dataset, launchPuppeteer } from 'crawlee';
+
+// Launch the web browser.
+const browser = await launchPuppeteer();
+
+// Create and navigate new page
+console.log('Open target page');
+const page = await browser.newPage();
+await page.goto('https://github.com/search/advanced');
+
+// Fill form fields and select desired search options
+console.log('Fill in search form');
+await page.type('#adv_code_search input.js-advanced-search-input', 'apify-js');
+await page.type('#search_from', 'apify');
+await page.type('#search_date', '>2015');
+await page.select('select#search_language', 'JavaScript');
+
+// Submit the form and wait for full load of next page
+console.log('Submit search form');
+await Promise.all([
+ page.waitForNavigation({ waitUntil: 'networkidle2' }),
+ page.click('#adv_code_search button[type="submit"]'),
+]);
+
+// Obtain and print list of search results
+const results = await page.$$eval('[data-testid="results-list"] div.search-title > a',
+ (nodes) => nodes.map((node) => ({
+ url: node.href,
+ name: node.innerText,
+ })),
+);
+
+console.log('Results:', results);
+
+// Store data in default dataset
+await Dataset.pushData(results);
+
+// Close browser
+await browser.close();
diff --git a/website/versioned_docs/version-3.7/examples/http_crawler.mdx b/website/versioned_docs/version-3.7/examples/http_crawler.mdx
new file mode 100644
index 000000000..e62793ae5
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/http_crawler.mdx
@@ -0,0 +1,14 @@
+---
+id: http-crawler
+title: HTTP crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';
+
+This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {HttpCrawlerSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/http_crawler.ts b/website/versioned_docs/version-3.7/examples/http_crawler.ts
new file mode 100644
index 000000000..4052df131
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/http_crawler.ts
@@ -0,0 +1,54 @@
+import { Dataset, HttpCrawler, log, LogLevel } from 'crawlee';
+
+// Crawlers come with various utilities, e.g. for logging.
+// Here we use debug level of logging to improve the debugging experience.
+// This functionality is optional!
+log.setLevel(LogLevel.DEBUG);
+
+// Create an instance of the HttpCrawler class - a crawler
+// that automatically loads the URLs and saves their HTML.
+const crawler = new HttpCrawler({
+ // The crawler downloads and processes the web pages in parallel, with a concurrency
+ // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // Here we define some hard limits for the concurrency.
+ minConcurrency: 10,
+ maxConcurrency: 50,
+
+ // On error, retry each page at most once.
+ maxRequestRetries: 1,
+
+ // Increase the timeout for processing of each page.
+ requestHandlerTimeoutSecs: 30,
+
+ // Limit to 10 requests per one crawl
+ maxRequestsPerCrawl: 10,
+
+ // This function will be called for each URL to crawl.
+ // It accepts a single parameter, which is an object with options as:
+ // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler
+ // We use for demonstration only 2 of them:
+ // - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method
+ // - body: the HTML code of the current page
+ async requestHandler({ request, body }) {
+ log.debug(`Processing ${request.url}...`);
+
+ // Store the results to the dataset. In local configuration,
+ // the data will be stored as JSON files in ./storage/datasets/default
+ await Dataset.pushData({
+ url: request.url,
+ body,
+ });
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries + 1 times.
+ failedRequestHandler({ request }) {
+ log.debug(`Request ${request.url} failed twice.`);
+ },
+});
+
+// Run the crawler and wait for it to finish.
+await crawler.run([
+ 'https://crawlee.dev',
+]);
+
+log.debug('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/jsdom_crawler.mdx b/website/versioned_docs/version-3.7/examples/jsdom_crawler.mdx
new file mode 100644
index 000000000..32732a0d8
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/jsdom_crawler.mdx
@@ -0,0 +1,22 @@
+---
+id: jsdom-crawler
+title: JSDOM crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import JSDOMCrawlerSource from '!!raw-loader!roa-loader!./jsdom_crawler.ts';
+import JSDOMCrawlerRunScriptSource from '!!raw-loader!roa-loader!./jsdom_crawler_react.ts';
+
+This example demonstrates how to use <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> to interact with a website using [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation.
+Here the script will open a calculator app from the [React examples](https://reactjs.org/community/examples.html), click `1` `+` `1` `=` and extract the result.
+
+<RunnableCodeBlock className="language-ts" type="cheerio">
+ {JSDOMCrawlerRunScriptSource}
+</RunnableCodeBlock>
+
+In the following example, we use <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, parse the HTML using the [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation and extract some data from it: the page title and all `h1` tags.
+
+<RunnableCodeBlock className="language-ts" type="cheerio">
+ {JSDOMCrawlerSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts
new file mode 100644
index 000000000..e72034113
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts
@@ -0,0 +1,64 @@
+import { Dataset, JSDOMCrawler, log, LogLevel } from 'crawlee';
+
+// Crawlers come with various utilities, e.g. for logging.
+// Here we use debug level of logging to improve the debugging experience.
+// This functionality is optional!
+log.setLevel(LogLevel.DEBUG);
+
+// Create an instance of the JSDOMCrawler class - a crawler
+// that automatically loads the URLs and parses their HTML using the jsdom library.
+const crawler = new JSDOMCrawler({
+ // The crawler downloads and processes the web pages in parallel, with a concurrency
+ // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // Here we define some hard limits for the concurrency.
+ minConcurrency: 10,
+ maxConcurrency: 50,
+
+ // On error, retry each page at most once.
+ maxRequestRetries: 1,
+
+ // Increase the timeout for processing of each page.
+ requestHandlerTimeoutSecs: 30,
+
+ // Limit to 10 requests per one crawl
+ maxRequestsPerCrawl: 10,
+
+ // This function will be called for each URL to crawl.
+ // It accepts a single parameter, which is an object with options as:
+ // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler
+ // We use for demonstration only 2 of them:
+ // - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method
+ // - window: the JSDOM window object
+ async requestHandler({ request, window }) {
+ log.debug(`Processing ${request.url}...`);
+
+ // Extract data from the page
+ const title = window.document.title;
+ const h1texts: { text: string }[] = [];
+ window.document.querySelectorAll('h1').forEach((element) => {
+ h1texts.push({
+ text: element.textContent!,
+ });
+ });
+
+ // Store the results to the dataset. In local configuration,
+ // the data will be stored as JSON files in ./storage/datasets/default
+ await Dataset.pushData({
+ url: request.url,
+ title,
+ h1texts,
+ });
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries + 1 times.
+ failedRequestHandler({ request }) {
+ log.debug(`Request ${request.url} failed twice.`);
+ },
+});
+
+// Run the crawler and wait for it to finish.
+await crawler.run([
+ 'https://crawlee.dev',
+]);
+
+log.debug('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/jsdom_crawler_react.ts b/website/versioned_docs/version-3.7/examples/jsdom_crawler_react.ts
new file mode 100644
index 000000000..8b3ec163b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/jsdom_crawler_react.ts
@@ -0,0 +1,32 @@
+import { JSDOMCrawler, log } from 'crawlee';
+
+// Create an instance of the JSDOMCrawler class - crawler that automatically
+// loads the URLs and parses their HTML using the jsdom library.
+const crawler = new JSDOMCrawler({
+ // Setting the `runScripts` option to `true` allows the crawler to execute client-side
+ // JavaScript code on the page. This is required for some websites (such as the React application in this example), but may pose a security risk.
+ runScripts: true,
+ // This function will be called for each crawled URL.
+ // Here we extract the window object from the options and use it to extract data from the page.
+ requestHandler: async ({ window }) => {
+ const { document } = window;
+ // The `document` object is analogous to the `window.document` object you know from your favourite web browsers.
+ // Thanks to this, you can use the regular browser-side APIs here.
+ document.querySelectorAll('button')[12].click(); // 1
+ document.querySelectorAll('button')[15].click(); // +
+ document.querySelectorAll('button')[12].click(); // 1
+ document.querySelectorAll('button')[18].click(); // =
+
+ const result = document.querySelectorAll('.component-display')[0].childNodes[0] as Element;
+ // The result is passed to the console. Unlike with Playwright or Puppeteer crawlers,
+ // this console call goes to the Node.js console, not the browser console. All the code here runs right in Node.js!
+ log.info(result.innerHTML); // 2
+ },
+});
+
+// Run the crawler and wait for it to finish.
+await crawler.run([
+ 'https://ahfarmer.github.io/calculator/',
+]);
+
+log.debug('Crawler finished.');
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.7/examples/map.ts b/website/versioned_docs/version-3.7/examples/map.ts
new file mode 100644
index 000000000..a06142a91
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/map.ts
@@ -0,0 +1,28 @@
+import { Dataset, KeyValueStore } from 'crawlee';
+
+const dataset = await Dataset.open<{
+ url: string,
+ headingCount: number,
+}>();
+
+// Seeding the dataset with some data
+await dataset.pushData([
+ {
+ url: 'https://crawlee.dev/',
+ headingCount: 11,
+ },
+ {
+ url: 'https://crawlee.dev/storage',
+ headingCount: 8,
+ },
+ {
+ url: 'https://crawlee.dev/proxy',
+ headingCount: 4,
+ },
+]);
+
+// Calling map function and filtering through mapped items...
+const moreThan5headers = (await dataset.map((item) => item.headingCount)).filter((count) => count > 5);
+
+// Saving the result of map to default key-value store...
+await KeyValueStore.setValue('pages_with_more_than_5_headers', moreThan5headers);
diff --git a/website/versioned_docs/version-3.7/examples/map_and_reduce.mdx b/website/versioned_docs/version-3.7/examples/map_and_reduce.mdx
new file mode 100644
index 000000000..46adb446a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/map_and_reduce.mdx
@@ -0,0 +1,80 @@
+---
+id: map-and-reduce
+title: Dataset Map and Reduce methods
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import MapSource from '!!raw-loader!roa-loader!./map.ts';
+import ReduceSource from '!!raw-loader!roa-loader!./reduce.ts';
+
+This example shows an easy use-case of the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> <ApiLink to="core/class/Dataset#map">`map`</ApiLink>
+and <ApiLink to="core/class/Dataset#reduce">`reduce`</ApiLink> methods. Both methods can be used to simplify
+the dataset results workflow process. Both can be called on the <ApiLink to="core/class/Dataset">dataset</ApiLink> directly.
+
+Important to mention is that both methods return a new result (`map` returns a new array and `reduce` can return any type) - neither method updates
+the dataset in any way.
+
+Examples for both methods are demonstrated on a simple dataset containing the results scraped from a page: the `URL` and a hypothetical number of
+`h1` - `h3` header elements under the `headingCount` key.
+
+This data structure is stored in the default dataset under `{PROJECT_FOLDER}/storage/datasets/default/`. If you want to simulate the
+functionality, you can use the <ApiLink to="core/class/Dataset#pushData">`dataset.pushData()`</ApiLink>
+method to save the example `JSON array` to your dataset.
+
+```json
+[
+ {
+ "url": "https://crawlee.dev/",
+ "headingCount": 11
+ },
+ {
+ "url": "https://crawlee.dev/storage",
+ "headingCount": 8
+ },
+ {
+ "url": "https://crawlee.dev/proxy",
+ "headingCount": 4
+ }
+]
+```
+
+### Map
+
+The dataset `map` method is very similar to standard Array mapping methods. It produces a new array of values by mapping each value in the existing
+array through a transformation function and an options parameter.
+
+The `map` method used to check if are there more than 5 header elements on each page:
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {MapSource}
+</RunnableCodeBlock>
+
+The `moreThan5headers` variable is an array of `headingCount` attributes where the number of headers is greater than 5.
+
+The `map` method's result value saved to the <ApiLink to="core/class/KeyValueStore">`key-value store`</ApiLink> should be:
+
+```javascript
+[11, 8]
+```
+
+### Reduce
+
+The dataset `reduce` method does not produce a new array of values - it reduces a list of values down to a single value. The method iterates through
+the items in the dataset using the <ApiLink to="core/class/Dataset#reduce">`memo` argument</ApiLink>. After performing the necessary
+calculation, the `memo` is sent to the next iteration, while the item just processed is reduced (removed).
+
+Using the `reduce` method to get the total number of headers scraped (all items in the dataset):
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {ReduceSource}
+</RunnableCodeBlock>
+
+The original dataset will be reduced to a single value, `pagesHeadingCount`, which contains the count of all headers for all scraped pages (all
+dataset items).
+
+The `reduce` method's result value saved to the <ApiLink to="core/class/KeyValueStore">`key-value store`</ApiLink> should be:
+
+```javascript
+23
+```
diff --git a/website/versioned_docs/version-3.7/examples/playwright_crawler.mdx b/website/versioned_docs/version-3.7/examples/playwright_crawler.mdx
new file mode 100644
index 000000000..c029c2b32
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/playwright_crawler.mdx
@@ -0,0 +1,22 @@
+---
+id: playwright-crawler
+title: Playwright crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./playwright_crawler.ts';
+
+This example demonstrates how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> in combination with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Playwright.
+
+The crawler starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./storage/datasets/default`.
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-playwright-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/playwright_crawler.ts b/website/versioned_docs/version-3.7/examples/playwright_crawler.ts
new file mode 100644
index 000000000..d27830b76
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/playwright_crawler.ts
@@ -0,0 +1,64 @@
+import { Dataset, PlaywrightCrawler } from 'crawlee';
+
+// Create an instance of the PlaywrightCrawler class - a crawler
+// that automatically loads the URLs in headless Chrome / Playwright.
+const crawler = new PlaywrightCrawler({
+ launchContext: {
+ // Here you can set options that are passed to the playwright .launch() function.
+ launchOptions: {
+ headless: true,
+ },
+ },
+
+ // Stop crawling after several pages
+ maxRequestsPerCrawl: 50,
+
+ // This function will be called for each URL to crawl.
+ // Here you can write the Playwright scripts you are familiar with,
+ // with the exception that browsers and pages are automatically managed by Crawlee.
+ // The function accepts a single parameter, which is an object with a lot of properties,
+ // the most important being:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - page: Playwright's Page object (see https://playwright.dev/docs/api/class-page)
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ log.info(`Processing ${request.url}...`);
+
+ // A function to be evaluated by Playwright within the browser context.
+ const data = await page.$$eval('.athing', ($posts) => {
+ const scrapedData: { title: string; rank: string; href: string }[] = [];
+
+ // We're getting the title, rank and URL of each post on Hacker News.
+ $posts.forEach(($post) => {
+ scrapedData.push({
+ title: $post.querySelector('.title a').innerText,
+ rank: $post.querySelector('.rank').innerText,
+ href: $post.querySelector('.title a').href,
+ });
+ });
+
+ return scrapedData;
+ });
+
+ // Store the results to the default dataset.
+ await Dataset.pushData(data);
+
+ // Find a link to the next page and enqueue it if it exists.
+ const infos = await enqueueLinks({
+ selector: '.morelink',
+ });
+
+ if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ failedRequestHandler({ request, log }) {
+ log.info(`Request ${request.url} failed too many times.`);
+ },
+});
+
+await crawler.addRequests(['https://news.ycombinator.com/']);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
+
+console.log('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.mdx b/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.mdx
new file mode 100644
index 000000000..08aa865f1
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.mdx
@@ -0,0 +1,22 @@
+---
+id: playwright-crawler-firefox
+title: Using Firefox browser with Playwright crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import PlaywrightFirefoxSource from '!!raw-loader!roa-loader!./playwright_crawler_firefox.ts';
+
+This example demonstrates how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> with headless Firefox browser.
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-playwright-firefox` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js">
+ {PlaywrightFirefoxSource}
+</RunnableCodeBlock>
+
+To see a real-world example of how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> in combination with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> to recursively scrape the [Hacker News website](https://news.ycombinator.com) check out the [`Playwright crawler example`](./playwright-crawler).
diff --git a/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.ts b/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.ts
new file mode 100644
index 000000000..cd91b2121
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/playwright_crawler_firefox.ts
@@ -0,0 +1,22 @@
+import { PlaywrightCrawler } from 'crawlee';
+import { firefox } from 'playwright';
+
+// Create an instance of the PlaywrightCrawler class.
+const crawler = new PlaywrightCrawler({
+ launchContext: {
+ // Set the Firefox browser to be used by the crawler.
+ // If launcher option is not specified here,
+ // default Chromium browser will be used.
+ launcher: firefox,
+ },
+ async requestHandler({ request, page, log }) {
+ const pageTitle = await page.title();
+
+ log.info(`URL: ${request.loadedUrl} | Page title: ${pageTitle}`);
+ },
+});
+
+await crawler.addRequests(['https://example.com']);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_capture_screenshot.mdx b/website/versioned_docs/version-3.7/examples/puppeteer_capture_screenshot.mdx
new file mode 100644
index 000000000..b9387e868
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_capture_screenshot.mdx
@@ -0,0 +1,75 @@
+---
+id: capture-screenshot
+title: Capture a screenshot using Puppeteer
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+
+import PuppeteerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer_page_screenshot.ts';
+import PuppeteerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_utils_snapshot.ts';
+import PuppeteerCrawlerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_page_screenshot.ts';
+import PuppeteerCrawlerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_crawler_utils_snapshot.ts';
+
+## Using Puppeteer directly
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+This example captures a screenshot of a web page using `Puppeteer`. It would look almost exactly the same with `Playwright`.
+
+<Tabs groupId="puppeteer-capture-screenshot">
+<TabItem value="pagescreenshot" label="Page Screenshot">
+
+Using `page.screenshot()`:
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerPageScreenshotSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="crawlerutilsscreenshot" label="Crawler Utils Screenshot" default>
+
+Using `utils.puppeteer.saveSnapshot()`:
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerCrawlerUtilsSnapshotSource}
+</RunnableCodeBlock>
+
+</TabItem>
+</Tabs>
+
+## Using `PuppeteerCrawler`
+
+This example captures a screenshot of multiple web pages when using `PuppeteerCrawler`:
+
+<Tabs groupId="puppeteer-capture-screenshot">
+<TabItem value="pagescreenshot" label="Page Screenshot">
+
+Using `page.screenshot()`:
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerCrawlerPageScreenshotSource}
+</RunnableCodeBlock>
+
+</TabItem>
+
+<TabItem value="crawlerutilsscreenshot" label="Crawler Utils Screenshot" default>
+
+Using the context-aware <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#saveSnapshot">`saveSnapshot()`</ApiLink> utility:
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {PuppeteerCrawlerCrawlerUtilsSnapshotSource}
+</RunnableCodeBlock>
+
+</TabItem>
+</Tabs>
+
+In both examples using `page.screenshot()`, a `key` variable is created based on the URL of the web page. This variable is used as the key when saving
+each screenshot into a key-value store.
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_crawler.mdx b/website/versioned_docs/version-3.7/examples/puppeteer_crawler.mdx
new file mode 100644
index 000000000..a2df07301
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_crawler.mdx
@@ -0,0 +1,25 @@
+---
+id: puppeteer-crawler
+title: Puppeteer crawler
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./puppeteer_crawler.ts';
+
+This example demonstrates how to use <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> in combination
+with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>
+to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Puppeteer.
+
+The crawler starts with a single URL, finds links to next pages, enqueues them and continues until no more desired links are available. The results
+are stored to the default dataset. In local configuration, the results are stored as JSON files in `./storage/datasets/default`
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_crawler.ts b/website/versioned_docs/version-3.7/examples/puppeteer_crawler.ts
new file mode 100644
index 000000000..17d74e1b4
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_crawler.ts
@@ -0,0 +1,64 @@
+import { Dataset, PuppeteerCrawler } from 'crawlee';
+
+// Create an instance of the PuppeteerCrawler class - a crawler
+// that automatically loads the URLs in headless Chrome / Puppeteer.
+const crawler = new PuppeteerCrawler({
+ // Here you can set options that are passed to the launchPuppeteer() function.
+ launchContext: {
+ launchOptions: {
+ headless: true,
+ // Other Puppeteer options
+ },
+ },
+
+ // Stop crawling after several pages
+ maxRequestsPerCrawl: 50,
+
+ // This function will be called for each URL to crawl.
+ // Here you can write the Puppeteer scripts you are familiar with,
+ // with the exception that browsers and pages are automatically managed by Crawlee.
+ // The function accepts a single parameter, which is an object with the following fields:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ log.info(`Processing ${request.url}...`);
+
+ // A function to be evaluated by Puppeteer within the browser context.
+ const data = await page.$$eval('.athing', ($posts) => {
+ const scrapedData: { title: string; rank: string; href: string }[] = [];
+
+ // We're getting the title, rank and URL of each post on Hacker News.
+ $posts.forEach(($post) => {
+ scrapedData.push({
+ title: $post.querySelector('.title a').innerText,
+ rank: $post.querySelector('.rank').innerText,
+ href: $post.querySelector('.title a').href,
+ });
+ });
+
+ return scrapedData;
+ });
+
+ // Store the results to the default dataset.
+ await Dataset.pushData(data);
+
+ // Find a link to the next page and enqueue it if it exists.
+ const infos = await enqueueLinks({
+ selector: '.morelink',
+ });
+
+ if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ failedRequestHandler({ request, log }) {
+ log.error(`Request ${request.url} failed too many times.`);
+ },
+});
+
+await crawler.addRequests(['https://news.ycombinator.com/']);
+
+// Run the crawler and wait for it to finish.
+await crawler.run();
+
+console.log('Crawler finished.');
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_crawler_crawler_utils_snapshot.ts b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_crawler_utils_snapshot.ts
new file mode 100644
index 000000000..fbc1fcc7a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_crawler_utils_snapshot.ts
@@ -0,0 +1,20 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+// Create a PuppeteerCrawler
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ request, saveSnapshot }) {
+ // Convert the URL into a valid key
+ const key = request.url.replace(/[:/]/g, '_');
+ // Capture the screenshot
+ await saveSnapshot({ key, saveHtml: false });
+ },
+});
+
+await crawler.addRequests([
+ { url: 'http://www.example.com/page-1' },
+ { url: 'http://www.example.com/page-2' },
+ { url: 'http://www.example.com/page-3' },
+]);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_crawler_page_screenshot.ts b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_page_screenshot.ts
new file mode 100644
index 000000000..7cc4095c4
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_page_screenshot.ts
@@ -0,0 +1,22 @@
+import { PuppeteerCrawler, KeyValueStore } from 'crawlee';
+
+// Create a PuppeteerCrawler
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ request, page }) {
+ // Capture the screenshot with Puppeteer
+ const screenshot = await page.screenshot();
+ // Convert the URL into a valid key
+ const key = request.url.replace(/[:/]/g, '_');
+ // Save the screenshot to the default key-value store
+ await KeyValueStore.setValue(key, screenshot, { contentType: 'image/png' });
+ },
+});
+
+await crawler.addRequests([
+ { url: 'http://www.example.com/page-1' },
+ { url: 'http://www.example.com/page-2' },
+ { url: 'http://www.example.com/page-3' },
+]);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_crawler_utils_snapshot.ts b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_utils_snapshot.ts
new file mode 100644
index 000000000..13f915091
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_crawler_utils_snapshot.ts
@@ -0,0 +1,17 @@
+import { launchPuppeteer, utils } from 'crawlee';
+
+const url = 'http://www.example.com/';
+// Start a browser
+const browser = await launchPuppeteer();
+
+// Open new tab in the browser
+const page = await browser.newPage();
+
+// Navigate to the URL
+await page.goto(url);
+
+// Capture the screenshot
+await utils.puppeteer.saveSnapshot(page, { key: 'my-key', saveHtml: false });
+
+// Close Puppeteer
+await browser.close();
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_page_screenshot.ts b/website/versioned_docs/version-3.7/examples/puppeteer_page_screenshot.ts
new file mode 100644
index 000000000..fd4e99c5f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_page_screenshot.ts
@@ -0,0 +1,22 @@
+import { KeyValueStore, launchPuppeteer } from 'crawlee';
+
+const keyValueStore = await KeyValueStore.open();
+
+const url = 'https://crawlee.dev';
+// Start a browser
+const browser = await launchPuppeteer();
+
+// Open new tab in the browser
+const page = await browser.newPage();
+
+// Navigate to the URL
+await page.goto(url);
+
+// Capture the screenshot
+const screenshot = await page.screenshot();
+
+// Save the screenshot to the default key-value store
+await keyValueStore.setValue('my-key', screenshot, { contentType: 'image/png' });
+
+// Close Puppeteer
+await browser.close();
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.mdx b/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.mdx
new file mode 100644
index 000000000..cd5b21c0b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.mdx
@@ -0,0 +1,20 @@
+---
+id: puppeteer-recursive-crawl
+title: Puppeteer recursive crawl
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import CrawlSource from '!!raw-loader!roa-loader!./puppeteer_recursive_crawl.ts';
+
+Run the following example to perform a recursive crawl of a website using <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
+
+:::tip
+
+To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.
+
+:::
+
+<RunnableCodeBlock className="language-js" type="puppeteer">
+ {CrawlSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.ts b/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.ts
new file mode 100644
index 000000000..ad48b3247
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/puppeteer_recursive_crawl.ts
@@ -0,0 +1,17 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.url}: ${title}`);
+
+ await enqueueLinks({
+ globs: ['http?(s)://www.iana.org/**'],
+ });
+ },
+ maxRequestsPerCrawl: 10,
+});
+
+await crawler.addRequests(['https://www.iana.org/']);
+
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/reduce.ts b/website/versioned_docs/version-3.7/examples/reduce.ts
new file mode 100644
index 000000000..243966c90
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/reduce.ts
@@ -0,0 +1,30 @@
+import { Dataset, KeyValueStore } from 'crawlee';
+
+const dataset = await Dataset.open<{
+ url: string,
+ headingCount: number,
+}>();
+
+// Seeding the dataset with some data
+await dataset.pushData([
+ {
+ url: 'https://crawlee.dev/',
+ headingCount: 11,
+ },
+ {
+ url: 'https://crawlee.dev/storage',
+ headingCount: 8,
+ },
+ {
+ url: 'https://crawlee.dev/proxy',
+ headingCount: 4,
+ },
+]);
+
+// calling reduce function and using memo to calculate number of headers
+const pagesHeadingCount = await dataset.reduce((memo, value) => {
+ return memo + value.headingCount;
+}, 0);
+
+// saving result of map to default Key-value store
+await KeyValueStore.setValue('pages_heading_count', pagesHeadingCount);
diff --git a/website/versioned_docs/version-3.7/examples/skip-navigation.mdx b/website/versioned_docs/version-3.7/examples/skip-navigation.mdx
new file mode 100644
index 000000000..5e6384020
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/skip-navigation.mdx
@@ -0,0 +1,23 @@
+---
+id: skip-navigation
+title: Skipping navigations for certain requests
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import SkipNavigationSource from '!!raw-loader!roa-loader!./skip-navigation.ts';
+
+While crawling a website, you may encounter certain resources you'd like to save, but don't need the full power of a crawler to do so (like images delivered through a CDN).
+
+By combining the <ApiLink to="core/class/Request#skipNavigation">`Request#skipNavigation`</ApiLink> option with <ApiLink to="basic-crawler/interface/BasicCrawlingContext#sendRequest">`sendRequest`</ApiLink>, we can fetch the image from the CDN, and save it to our key-value store without needing to use the full crawler.
+
+
+:::info
+
+For this example, we are using the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to showcase this, but this is available on all the crawlers we provide.
+
+:::
+
+<RunnableCodeBlock className="language-js">
+ {SkipNavigationSource}
+</RunnableCodeBlock>
diff --git a/website/versioned_docs/version-3.7/examples/skip-navigation.ts b/website/versioned_docs/version-3.7/examples/skip-navigation.ts
new file mode 100644
index 000000000..0bbde53c1
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/skip-navigation.ts
@@ -0,0 +1,31 @@
+import { PlaywrightCrawler, KeyValueStore } from 'crawlee';
+
+// Create a key value store for all images we find
+const imageStore = await KeyValueStore.open('images');
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ request, page, sendRequest }) {
+ // The request should have the navigation skipped
+ if (request.skipNavigation) {
+ // Request the image and get its buffer back
+ const imageResponse = await sendRequest({ responseType: 'buffer' });
+
+ // Save the image in the key-value store
+ await imageStore.setValue(`${request.userData.key}.png`, imageResponse.body);
+
+ // Prevent executing the rest of the code as we do not need it
+ return;
+ }
+
+ // Get all the image sources in the current page
+ const images = await page.$$eval('img', (imgs) => imgs.map((img) => img.src));
+
+ // Add all the urls as requests for the crawler, giving each image a key
+ await crawler.addRequests(images.map((url, i) => ({ url, skipNavigation: true, userData: { key: i } })));
+ },
+});
+
+await crawler.addRequests(['https://crawlee.dev']);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/examples/tsconfig.json b/website/versioned_docs/version-3.7/examples/tsconfig.json
new file mode 100644
index 000000000..62b847082
--- /dev/null
+++ b/website/versioned_docs/version-3.7/examples/tsconfig.json
@@ -0,0 +1,21 @@
+{
+ "extends": "@apify/tsconfig",
+ "compilerOptions": {
+ "baseUrl": "../..",
+ "paths": {
+ "crawlee": ["packages/crawlee/src"],
+ "@crawlee/basic": ["packages/basic-crawler/src"],
+ "@crawlee/browser": ["packages/browser-crawler/src"],
+ "@crawlee/http": ["packages/http-crawler/src"],
+ "@crawlee/linkedom": ["packages/linkedom-crawler/src"],
+ "@crawlee/jsdom": ["packages/jsdom-crawler/src"],
+ "@crawlee/cheerio": ["packages/cheerio-crawler/src"],
+ "@crawlee/playwright": ["packages/playwright-crawler/src"],
+ "@crawlee/puppeteer": ["packages/puppeteer-crawler/src"],
+ "@crawlee/*": ["packages/*/src"]
+ },
+ "target": "ES2022",
+ "module": "ES2022",
+ "noUnusedParameters": false
+ }
+}
diff --git a/website/versioned_docs/version-3.7/experiments/request_locking.mdx b/website/versioned_docs/version-3.7/experiments/request_locking.mdx
new file mode 100644
index 000000000..30c9cd74d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/experiments/request_locking.mdx
@@ -0,0 +1,131 @@
+---
+id: experiments-request-locking
+title: Request Locking
+description: Parallelize crawlers with ease using request locking
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+:::caution
+
+This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.
+
+The API is subject to change, and we might introduce breaking changes in the future.
+
+Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.
+
+:::
+
+Starting with `crawlee` version `3.5.5`, we have introduced a new crawler option that lets you enable using a new request locking
+API. With this API, you will be able to pass a `RequestQueue` to multiple crawlers to parallelize the crawling process.
+
+:::info Keep in mind
+
+The request queue that supports request locking is currently exported via the `RequestQueue2` class. Once the experiment is over, this class will replace
+the current `RequestQueue` class
+
+:::
+
+## How to enable the experiment
+
+### In crawlers
+
+:::note
+
+This example shows how to enable the experiment in the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>,
+but you can apply this to any crawler type.
+
+:::
+
+```ts
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // highlight-next-line
+ experiments: {
+ // highlight-next-line
+ requestLocking: true,
+ // highlight-next-line
+ },
+ async requestHandler({ $, request }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
+```
+
+### Outside crawlers (to setup your own request queue that supports locking)
+
+Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueue2` instead.
+
+```ts
+// highlight-next-line
+import { RequestQueue2 } from 'crawlee';
+
+const queue = await RequestQueue2.open('my-locking-queue');
+await queue.addRequests([
+ { url: 'https://crawlee.dev' },
+ { url: 'https://crawlee.dev/docs' },
+ { url: 'https://crawlee.dev/api' },
+]);
+```
+
+### Using the new request queue in crawlers
+
+If you make your own request queue that supports locking, you will also need to enable the experiment in your crawlers.
+
+:::danger
+
+If you do not enable the experiment, you will receive a runtime error and the crawler will not start.
+
+:::
+
+```ts
+import { CheerioCrawler, RequestQueue2 } from 'crawlee';
+
+// highlight-next-line
+const queue = await RequestQueue2.open('my-locking-queue');
+
+const crawler = new CheerioCrawler({
+ // highlight-next-line
+ experiments: {
+ // highlight-next-line
+ requestLocking: true,
+ // highlight-next-line
+ },
+ // highlight-next-line
+ requestQueue: queue,
+ async requestHandler({ $, request }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ },
+});
+
+await crawler.run();
+```
+
+## Other changes
+
+:::info
+
+This section is only useful if you're a tinkerer and want to see what's going on under the hood.
+
+:::
+
+In order to facilitate the new request locking API, as well as keep both the current request queue logic and the new, locking based request queue
+logic, we have implemented a common starting point called `RequestProvider`.
+
+This class implements almost all functions by default, but expects you, the developer, to implement the following methods:
+`fetchNextRequest` and `ensureHeadIsNotEmpty`. These methods are responsible for loading and returning requests to process,
+and tell crawlers if there are more requests to process.
+
+You can use this base class to implement your own request providers if you need to fetch requests from a different source.
+
+:::tip
+
+We recommend you use TypeScript when implementing your own request provider, as it comes with suggestions for the abstract methods, as well as
+giving you the exact types you need to return.
+
+:::
diff --git a/website/versioned_docs/version-3.7/guides/avoid_blocking.mdx b/website/versioned_docs/version-3.7/guides/avoid_blocking.mdx
new file mode 100644
index 000000000..e23cad57f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/avoid_blocking.mdx
@@ -0,0 +1,61 @@
+---
+id: avoid-blocking
+title: Avoid getting blocked
+description: How to avoid getting blocked when scraping
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import PlaywrightSource from '!!raw-loader!./avoid_blocking_playwright.ts';
+import PuppeteerSource from '!!raw-loader!./avoid_blocking_puppeteer';
+import PlaywrightFingerprintsOffSource from '!!raw-loader!./avoid_blocking_playwright_fingerprints_off.ts';
+import PuppeteerFingerprintsOffSource from '!!raw-loader!./avoid_blocking_puppeteer_fingerprints_off.ts';
+
+A scraper might get blocked for numerous reasons. Let's narrow it down to the two main ones. The first is a bad or blocked IP address. You can learn about this topic in the [proxy management guide](./proxy-management). The second reason is [browser fingerprints](https://pixelprivacy.com/resources/browser-fingerprinting/) (or signatures), which we will explore more in this guide. Check the [Apify Academy anti-scraping course](https://docs.apify.com/academy/anti-scraping) to gain a deeper theoretical understanding of blocking and learn a few tips and tricks.
+
+Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.
+
+## Using browser fingerprints
+
+Changing browser fingerprints can be a tedious job. Luckily, Crawlee provides this feature with zero configuration necessary - the usage of fingerprints is enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler`. So whenever we build a scraper that is using one of these crawlers - the fingerprints are going to be generated for the default browser and the operating system out of the box.
+
+## Customizing browser fingerprints
+
+In certain cases we want to narrow down the fingerprints used - e.g. specify a certain operating system, locale or browser. This is also possible with Crawlee - the crawler can have the generation algorithm customized to reflect the particular browser version and many more. Let's take a look at the examples bellow:
+
+<Tabs groupId="avoid_blocking">
+ <TabItem value="playwright" label="PlaywrightCrawler" default>
+ <CodeBlock language="js">
+ {PlaywrightSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {PuppeteerSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+## Disabling browser fingerprints
+
+On the contrary, sometimes we want to entirely disable the usage of browser fingerprints. This is easy to do with Crawlee too. All we have to do is set the `useFingerprints` option of the `browserPoolOptions` to `false`:
+
+<Tabs groupId="avoid_blocking">
+ <TabItem value="playwright" label="PlaywrightCrawler" default>
+ <CodeBlock language="js">
+ {PlaywrightFingerprintsOffSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {PuppeteerFingerprintsOffSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+**Related links**
+
+- [Fingerprint Suite Docs](https://github.com/apify/fingerprint-suite)
+- [Apify Academy anti-scraping course](https://docs.apify.com/academy/anti-scraping)
diff --git a/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright.ts b/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright.ts
new file mode 100644
index 000000000..8a8543504
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright.ts
@@ -0,0 +1,23 @@
+import { PlaywrightCrawler } from 'crawlee';
+import { BrowserName, DeviceCategory, OperatingSystemsName } from '@crawlee/browser-pool';
+
+const crawler = new PlaywrightCrawler({
+ browserPoolOptions: {
+ useFingerprints: true, // this is the default
+ fingerprintOptions: {
+ fingerprintGeneratorOptions: {
+ browsers: [{
+ name: BrowserName.edge,
+ minVersion: 96,
+ }],
+ devices: [
+ DeviceCategory.desktop,
+ ],
+ operatingSystems: [
+ OperatingSystemsName.windows,
+ ],
+ },
+ },
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright_fingerprints_off.ts b/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright_fingerprints_off.ts
new file mode 100644
index 000000000..c36e52fc3
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/avoid_blocking_playwright_fingerprints_off.ts
@@ -0,0 +1,8 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ browserPoolOptions: {
+ useFingerprints: false,
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer.ts b/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer.ts
new file mode 100644
index 000000000..6a4a2ea2e
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer.ts
@@ -0,0 +1,23 @@
+import { PuppeteerCrawler } from 'crawlee';
+import { BrowserName, DeviceCategory } from '@crawlee/browser-pool';
+
+const crawler = new PuppeteerCrawler({
+ browserPoolOptions: {
+ useFingerprints: true, // this is the default
+ fingerprintOptions: {
+ fingerprintGeneratorOptions: {
+ browsers: [
+ BrowserName.chrome,
+ BrowserName.firefox,
+ ],
+ devices: [
+ DeviceCategory.mobile,
+ ],
+ locales: [
+ 'en-US',
+ ],
+ },
+ },
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer_fingerprints_off.ts b/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer_fingerprints_off.ts
new file mode 100644
index 000000000..425b4d3d5
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/avoid_blocking_puppeteer_fingerprints_off.ts
@@ -0,0 +1,8 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ browserPoolOptions: {
+ useFingerprints: false,
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/cheerio_crawler.mdx b/website/versioned_docs/version-3.7/guides/cheerio_crawler.mdx
new file mode 100644
index 000000000..6097f77b4
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/cheerio_crawler.mdx
@@ -0,0 +1,87 @@
+---
+id: cheerio-crawler-guide
+title: "CheerioCrawler guide"
+sidebar_label: "CheerioCrawler"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+&#8203;<ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> is our simplest and fastest crawler. If you're familiar with [jQuery](https://jquery.com/), you'll understand `CheerioCrawler` in minutes.
+
+## What is Cheerio
+
+[Cheerio](https://www.npmjs.com/package/cheerio) is essentially [jQuery](https://jquery.com/) for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would use jQuery for manipulating the DOM of an HTML page. In crawling, you'll mostly use it to select the needed elements and extract their values - the data you're interested in. But jQuery runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> comes in.
+
+## How the crawler works
+
+&#8203;<ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client. The URLs are fed to the crawler using <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#additionalMimeTypes">`additionalMimeTypes`</ApiLink> option.
+
+:::info
+
+Modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards, and together they create the final page. To crawl those, see <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.
+
+:::
+
+Once the page's HTML is retrieved, the crawler will pass it to [Cheerio](https://www.npmjs.com/package/cheerio) for parsing. The result is the typical `$` function, which should be familiar to jQuery users. You can use the `$` function to do all sorts of lookups and manipulation of the page's HTML, but in scraping, you will mostly use it to find specific HTML elements and extract their data.
+
+Example use of Cheerio and its `$` function in comparison to browser JavaScript:
+
+```ts
+// Return the text content of the <title> element.
+document.querySelector('title').textContent; // plain JS
+$('title').text(); // Cheerio
+
+// Return an array of all 'href' links on the page.
+Array.from(document.querySelectorAll('[href]')).map(el => el.href); // plain JS
+$('[href]')
+ .map((i, el) => $(el).attr('href'))
+ .get(); // Cheerio
+```
+
+:::note
+
+This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides. Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is your best bet to do the parsing in Node.
+
+:::
+
+## When to use `CheerioCrawler`
+
+`CheerioCrawler` really shines when you need to cope with extremely high workloads. With just 4 GBs of memory and a single CPU core, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_. To scrape this fast with a full browser scraper, such as the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, you'd need significantly more computing power.
+
+**Advantages:**
+
+- Extremely fast and cheap to run
+- Easy to set up
+- Familiar for jQuery users
+- Automatically avoids some anti-scraping bans
+
+**Disadvantages:**
+
+- Does not work for websites that require JavaScript rendering
+- May easily overload the target website with requests
+- Does not enable any manipulation of the website before scraping
+
+## Example use of Cheerio
+
+### Get text content of an element
+
+Finds the first `<h2>` element and returns its text content.
+
+```js
+$('h2').text()
+```
+
+### Find all links on a page
+
+This snippet finds all `<a>` elements which have the `href` attribute and extracts the hrefs into an array.
+
+```js
+$('a[href]')
+ .map((i, el) => $(el).attr('href'))
+ .get();
+```
+
+### Other examples
+
+Visit the [Examples](../examples) section to browse examples of `CheerioCrawler` usage. Almost all examples show `CheerioCrawler` code in their code tabs.
diff --git a/website/versioned_docs/version-3.7/guides/configuration.mdx b/website/versioned_docs/version-3.7/guides/configuration.mdx
new file mode 100644
index 000000000..f5977e330
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/configuration.mdx
@@ -0,0 +1,224 @@
+---
+id: configuration
+title: Configuration
+description: Configuring Crawlee parameters
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+&#8203;<ApiLink to="core/class/Configuration">`Configuration`</ApiLink> is a class holding Crawlee configuration parameters. By default, you don't need to set or change any of them, but for certain use cases you might want to do so, e.g. in order to change the default storage directory, or enable verbose error logging, and so on.
+
+There are three ways of changing the configuration parameters:
+
+- adding `crawlee.json` file to your project
+- setting environment variables
+- using the `Configuration` class
+
+You could also combine all the above, but you should keep in mind, that the precedence for these 3 options is the following:
+***`crawlee.json`*** < ***constructor options*** < ***environment variables***.
+
+`crawlee.json` is a baseline. The options provided in the `Configuration` constructor will override the options provided in the JSON. Environment variables will override both.
+
+## `crawlee.json`
+
+The first option you could use for configuring Crawlee is `crawlee.json` file. The only thing you need to do is specify the <ApiLink to="core/interface/ConfigurationOptions">`ConfigurationOptions`</ApiLink> in the file, place the file in the root of your project, and Crawlee will use provided options as global configuration.
+
+```json title="crawlee.json"
+{
+ "persistStateIntervalMillis": 10000,
+ "logLevel": "DEBUG"
+}
+```
+
+With `crawlee.json` you don't need to do anything else in the code:
+
+```js
+import { CheerioCrawler, sleep } from 'crawlee';
+// We are not importing nor passing
+// the Configuration to the crawler.
+// We are not assigning any env vars either.
+const crawler = new CheerioCrawler();
+
+crawler.router.addDefaultHandler(async ({ request }) => {
+ // for the first request we wait for 5 seconds,
+ // and add the second request to the queue
+ if (request.url === 'https://www.example.com/1') {
+ await sleep(5_000);
+ await crawler.addRequests(['https://www.example.com/2'])
+ }
+ // for the second request we wait for 10 seconds,
+ // and abort the run
+ if (request.url === 'https://www.example.com/2') {
+ await sleep(10_000);
+ process.exit(0);
+ }
+});
+
+await crawler.run(['https://www.example.com/1']);
+```
+
+If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store,
+which would show, that there's 1 finished request and crawler runtime was ~10 seconds.
+This confirms that the state was persisted after 10 seconds, as it was set in `crawlee.json`.
+Besides, you should see `DEBUG` logs in addition to `INFO` ones in your terminal, as `logLevel` was set to `DEBUG` in the `crawlee.json`, meaning Crawlee picked both provided options correctly.
+
+## Environment Variables
+
+Another way of configuring Crawlee is setting environment variables.
+The following is a list of the environment variables used by Crawlee that are available to the user.
+
+### Important env vars
+
+The following environment variables have large impact on the way Crawlee works and its behavior
+can be changed significantly by setting or unsetting them.
+
+#### `CRAWLEE_STORAGE_DIR`
+
+Defines the path to a local directory where <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>, <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>, and <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> store their data. By default, it is set to `./storage`.
+
+#### `CRAWLEE_DEFAULT_DATASET_ID`
+
+The default dataset has ID `default`. Setting this environment variable overrides the default dataset ID with the provided value.
+
+#### `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID`
+
+The default key-value store has ID `default`. Setting this environment variable overrides the default key-value store ID with the provided value.
+
+#### `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID`
+
+The default request queue has ID `default`. Setting this environment variable overrides the default request queue ID with the provided value.
+
+#### `CRAWLEE_PURGE_ON_START`
+
+Storage directories are purged by default. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).
+
+### Convenience env vars
+
+The next group includes env vars that can help achieve certain goals without having to change
+our code, such as temporarily switching log level to DEBUG or enabling verbose logging for errors.
+
+#### `CRAWLEE_HEADLESS`
+
+If set to `1`, web browsers launched by Crawlee will run in the headless mode. We can still override
+this setting in the code, e.g. by passing the `headless: true` option to the <ApiLink to="puppeteer-crawler/function/launchPuppeteer">`launchPuppeteer()`</ApiLink> function. By default, the browsers
+are launched in headful mode, i.e. with windows.
+
+#### `CRAWLEE_LOG_LEVEL`
+
+Specifies the minimum log level, which can be one of the following values (in order of severity):
+`DEBUG`, `INFO`, `WARNING`, `ERROR` and `OFF`. By default, the log level is set to `INFO`,
+which means that `DEBUG` messages are not printed to console. See the <ApiLink to="core/class/Log">`utils.log`</ApiLink>
+namespace for logging utilities.
+
+#### `CRAWLEE_VERBOSE_LOG`
+
+Enables verbose logging if set to `true`. If not explicitly set to `true` - for errors thrown from inside request handler a warning with only error message will be logged as long as we know the request will be retried. Same applies to some known errors (such as timeout errors). Disabled by default.
+
+#### `CRAWLEE_MEMORY_MBYTES`
+
+Sets the amount of system memory in megabytes to be used by the <ApiLink to="core/class/AutoscaledPool">`AutoscaledPool`</ApiLink>.
+It is used to limit the number of concurrently running tasks. By default, the max amount of memory
+to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory,
+the autoscaling feature will only use up to 2048 MB of memory.
+
+## Configuration class
+
+The last option to adjust Crawlee configuration is to use the <ApiLink to="core/class/Configuration">`Configuration`</ApiLink> class in the code.
+
+### Global Configuration
+
+By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can get access to it via <ApiLink to="core/class/Configuration#getGlobalConfig">`Configuration.getGlobalConfig()`</ApiLink> function. Now you can easily <ApiLink to="core/class/Configuration#get">`get`</ApiLink> and <ApiLink to="core/class/Configuration#set">`set`</ApiLink> the <ApiLink to="core/interface/ConfigurationOptions">`ConfigurationOptions`</ApiLink>.
+
+```js
+import { CheerioCrawler, Configuration, sleep } from 'crawlee';
+
+// Get the global configuration
+const config = Configuration.getGlobalConfig();
+// Set the 'persistStateIntervalMillis' option
+// of global configuration to 10 seconds
+config.set('persistStateIntervalMillis', 10_000);
+
+// Note, that we are not passing the configuration to the crawler
+// as it's using the global configuration
+const crawler = new CheerioCrawler();
+
+crawler.router.addDefaultHandler(async ({ request }) => {
+ // For the first request we wait for 5 seconds,
+ // and add the second request to the queue
+ if (request.url === 'https://www.example.com/1') {
+ await sleep(5_000);
+ await crawler.addRequests(['https://www.example.com/2'])
+ }
+ // For the second request we wait for 10 seconds,
+ // and abort the run
+ if (request.url === 'https://www.example.com/2') {
+ await sleep(10_000);
+ process.exit(0);
+ }
+});
+
+await crawler.run(['https://www.example.com/1']);
+```
+
+This is pretty much the same example we used for showing `crawlee.json` usage,
+but now we're using the global configuration, which is the only difference.
+If you run this example - you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store as before,
+which would show the same number of finishes requests (one) and the same crawler runtime (~10 seconds).
+This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the global configuration.
+
+:::note
+
+After running the same example with commented two lines of code related to `Configuration` there will be
+no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store:
+as we did not change the `persistStateIntervalMillis`, Crawlee used the default value of 60 seconds,
+and the crawler was forcefully aborted after ~15 seconds of run time before it persisted the state for the first time.
+
+:::
+
+### Custom configuration
+
+Alternatively, you can create a custom configuration. In this case you need to pass it to the class that is going to use it, e.g. to the crawler. Let's adjust the previous example:
+
+```js
+import { CheerioCrawler, Configuration, sleep } from 'crawlee';
+
+// Create new configuration
+const config = new Configuration({
+ // Set the 'persistStateIntervalMillis' option to 10 seconds
+ persistStateIntervalMillis: 10_000,
+});
+
+// Now we need to pass the configuration to the crawler
+const crawler = new CheerioCrawler({}, config);
+
+crawler.router.addDefaultHandler(async ({ request }) => {
+ // for the first request we wait for 5 seconds,
+ // and add the second request to the queue
+ if (request.url === 'https://www.example.com/1') {
+ await sleep(5_000);
+ await crawler.addRequests(['https://www.example.com/2'])
+ }
+ // for the second request we wait for 10 seconds,
+ // and abort the run
+ if (request.url === 'https://www.example.com/2') {
+ await sleep(10_000);
+ process.exit(0);
+ }
+});
+
+await crawler.run(['https://www.example.com/1']);
+```
+
+If you run this example - it would work exactly the same as before,
+with the same `SDK_CRAWLER_STATISTICS` file in default Key-Value store after the run,
+showing the same number of finished requests and the same crawler run time.
+
+:::note
+
+If you would not pass the configuration to the crawler, there again will be
+no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.
+Since we did not pass the configuration to the crawler,
+the crawler will use the global configuration, which is using the default `persistStateIntervalMillis`.
+So again, the run was aborted before the state was persisted for the first time.
+
+:::
diff --git a/website/versioned_docs/version-3.7/guides/docker_browser_js.txt b/website/versioned_docs/version-3.7/guides/docker_browser_js.txt
new file mode 100644
index 000000000..afe996891
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/docker_browser_js.txt
@@ -0,0 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node-playwright-chrome:16
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt
new file mode 100644
index 000000000..e6477ffc3
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt
@@ -0,0 +1,51 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node-playwright-chrome:16 AS builder
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY --chown=myuser . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
+
+# Create final image
+FROM apify/actor-node-playwright-chrome:16
+
+# Copy only built JS files from builder image
+COPY --from=builder --chown=myuser /home/myuser/dist ./dist
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image. If you know you won't need headful browsers,
+# you can remove the XVFB start script for a micro perf gain.
+CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod --silent
diff --git a/website/versioned_docs/version-3.7/guides/docker_images.mdx b/website/versioned_docs/version-3.7/guides/docker_images.mdx
new file mode 100644
index 000000000..fc8f9480d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/docker_images.mdx
@@ -0,0 +1,188 @@
+---
+id: docker-images
+title: Running in Docker
+description: Example Docker images to run your crawlers
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import jsNodeDocker from '!!raw-loader!./docker_node_js.txt';
+import tsNodeDocker from '!!raw-loader!./docker_node_ts.txt';
+import jsBrowserDocker from '!!raw-loader!./docker_browser_js.txt';
+import tsBrowserDocker from '!!raw-loader!./docker_browser_ts.txt';
+
+Running headless browsers in Docker requires a lot of setup to do it right. But there's no need to worry about that, because we already created base images that you can freely use. We use them every day on the [Apify Platform](../deployment/apify-platform).
+
+All images can be found in their [GitHub repo](https://github.com/apify/apify-actor-docker) and in our [DockerHub](https://hub.docker.com/orgs/apify).
+
+## Overview
+
+Browsers are pretty big, so we try to provide a wide variety of images to suit the specific needs. Here's a full list of our Docker images.
+
+- [`apify/actor-node`](#actor-node)
+- [`apify/actor-node-puppeteer-chrome`](#actor-node-puppeteer-chrome)
+- [`apify/actor-node-playwright`](#actor-node-playwright)
+- [`apify/actor-node-playwright-chrome`](#actor-node-playwright-chrome)
+- [`apify/actor-node-playwright-firefox`](#actor-node-playwright-firefox)
+- [`apify/actor-node-playwright-webkit`](#actor-node-playwright-webkit)
+
+## Versioning
+
+Each image is tagged with up to 2 version tags, depending on the type of the image. One for Node.js version and second for pre-installed web automation library version. If you use the image name without a version tag, you'll always get the latest available version.
+
+> We recommend always using at least the Node.js version tag in production Dockerfiles. It will ensure that a future update of Node.js will not break our automations.
+
+### Node.js versioning
+
+Our images are built with multiple Node.js versions to ensure backwards compatibility. Currently, Node.js **versions 16 and 18 are supported** (legacy versions still exist, see DockerHub). To select the preferred version, use the appropriate number as the image tag.
+
+```dockerfile
+# Use Node.js 16
+FROM apify/actor-node:16
+```
+
+### Automation library versioning
+
+Images that include a pre-installed automation library, which means all images that include `puppeteer` or `playwright` in their name, are also tagged with the pre-installed version of the library. For example, `apify/actor-node-puppeteer-chrome:16-13.7.0` comes with Node.js 16 and Puppeteer v13.7.0. If you try to install a different version of Puppeteer into this image, you may run into compatibility issues, because the Chromium version bundled with `puppeteer` will not match the version of Chromium that's pre-installed.
+
+Similarly `apify/actor-node-playwright-firefox:14-1.21.1` runs on Node.js 14 and is pre-installed with the Firefox version that comes with v1.21.1.
+
+Installing `apify/actor-node-puppeteer-chrome` (without a tag) will install the latest available version of Node.js and `puppeteer`.
+
+### Pre-release tags
+
+We also build pre-release versions of the images to test the changes we make. Those are typically denoted by a `beta` suffix, but it can vary depending on our needs. If you need to try a pre-release version, you can do it like this:
+
+```dockerfile
+# Without library version.
+FROM apify/actor-node:16-beta
+```
+
+```dockerfile
+# With library version.
+FROM apify/actor-node-playwright-chrome:16-1.10.0-beta
+```
+
+## Best practices
+
+- Node.js version tag should **always** be used.
+- The automation library version tag should be used for **added security**.
+- Asterisk `*` should be used as the automation library version in our `package.json` files.
+
+It makes sure the pre-installed version of Puppeteer or Playwright is not re-installed on build. This is important, because those libraries are only guaranteed to work with specific versions of browsers, and those browsers come pre-installed in the image.
+
+```dockerfile
+FROM apify/actor-node-playwright-chrome:16
+```
+
+```json
+{
+ "dependencies": {
+ "crawlee": "^3.0.0",
+ "playwright": "*"
+ }
+}
+```
+
+### Warning about image size
+
+Browsers are huge. If you don't need them all in your image, it's better to use a smaller image with only the one browser you need.
+
+You should also be careful when installing new dependencies. Nothing prevents you from installing Playwright into the`actor-node-puppeteer-chrome` image, but the resulting image will be about 3 times larger and extremely slow to download and build.
+
+When you use only what you need, you'll be rewarded with reasonable build and start times.
+
+## Apify Docker Images
+
+### actor-node
+
+This is the smallest image we have based on Alpine Linux. It does not include any browsers, and it's therefore best used with <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>. It benefits from lightning fast builds and container startups.
+
+&#8203;<ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and other browser based features will **NOT** work with this image.
+
+```dockerfile
+FROM apify/actor-node:16
+```
+
+### actor-node-puppeteer-chrome
+
+This image includes Puppeteer (Chromium) and the Chrome browser. It can be used with <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>, but **NOT** with <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.
+
+The image supports XVFB by default, so you can run both `headless` and `headful` browsers with it.
+
+```dockerfile
+FROM apify/actor-node-puppeteer-chrome:16
+```
+
+### actor-node-playwright
+
+A very large and slow image that can run all Playwright browsers: Chromium, Chrome, Firefox, WebKit. Everything is installed. If you need to develop or test with multiple browsers, this is the image to choose, but in most cases, it's better to use the specialized images below.
+
+```dockerfile
+FROM apify/actor-node-playwright:16
+```
+
+### actor-node-playwright-chrome
+
+Similar to [`actor-node-puppeteer-chrome`](#actor-node-puppeteer-chrome), but for Playwright. You can run <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, but **NOT** <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
+
+It uses the [`PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD`](https://playwright.dev/docs/api/environment-variables/) environment variable to block installation of more browsers into the image to keep it small. If you want more browsers, either use the [`actor-node-playwright`](#actor-node-playwright) image override this env var.
+
+The image supports XVFB by default, so we can run both `headless` and `headful` browsers with it.
+
+```dockerfile
+FROM apify/actor-node-playwright-chrome:16
+```
+
+### actor-node-playwright-firefox
+
+Same idea as [`actor-node-playwright-chrome`](#actor-node-playwright-chrome), but with Firefox
+pre-installed.
+
+```dockerfile
+FROM apify/actor-node-playwright-firefox:16
+```
+
+### actor-node-playwright-webkit
+
+Same idea as [`actor-node-playwright-chrome`](#actor-node-playwright-chrome), but with WebKit
+pre-installed.
+
+```dockerfile
+FROM apify/actor-node-playwright-webkit:16
+```
+
+## Example Dockerfile
+
+To use the above images, it's necessary to have a [`Dockerfile`](https://docs.docker.com/engine/reference/builder/). You can either use this example, or bootstrap your projects with the [Crawlee CLI](../introduction/setting-up) which automatically adds the correct Dockerfile into our project folder.
+
+<Tabs>
+ <TabItem value="node+js" label="Node+JavaScript">
+ <CodeBlock language="dockerfile">
+ {jsNodeDocker}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="node+ts" label="Node+TypeScript">
+ <CodeBlock language="dockerfile">
+ {tsNodeDocker}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="browser+js" label="Browser+JavaScript">
+ This example is for Playwright. If you want to use Puppeteer, simply replace <b>playwright</b>
+ with <b>puppeteer</b> in the <code>FROM</code> declaration.
+ <CodeBlock language="dockerfile">
+ {jsBrowserDocker}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="browser+ts" label="Browser+TypeScript">
+ This example is for Playwright. If you want to use Puppeteer, simply replace <b>playwright</b>
+ with <b>puppeteer</b> in both <code>FROM</code> declarations.
+ <CodeBlock language="dockerfile">
+ {tsBrowserDocker}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/guides/docker_node_js.txt b/website/versioned_docs/version-3.7/guides/docker_node_js.txt
new file mode 100644
index 000000000..4018d6b99
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/docker_node_js.txt
@@ -0,0 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node:16
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/website/versioned_docs/version-3.7/guides/docker_node_ts.txt b/website/versioned_docs/version-3.7/guides/docker_node_ts.txt
new file mode 100644
index 000000000..43815f97d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/docker_node_ts.txt
@@ -0,0 +1,50 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node:16 AS builder
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
+
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
+
+# Create final image
+FROM apify/actor-node:16
+
+# Copy only built JS files from builder image
+COPY --from=builder /usr/src/app/dist ./dist
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY . ./
+
+
+# Run the image.
+CMD npm run start:prod --silent
diff --git a/website/versioned_docs/version-3.7/guides/got_scraping.mdx b/website/versioned_docs/version-3.7/guides/got_scraping.mdx
new file mode 100644
index 000000000..a67967210
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/got_scraping.mdx
@@ -0,0 +1,190 @@
+---
+id: got-scraping
+title: Got Scraping
+description: Blazing fast cURL alternative for modern web scraping
+---
+
+## Intro
+
+When using `BasicCrawler`, we have to send the requests manually. In order to do this, we can use the context-aware `sendRequest()` function:
+
+```ts
+import { BasicCrawler } from 'crawlee';
+
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ const res = await sendRequest();
+ log.info('received body', res.body);
+ },
+});
+```
+
+It uses [`got-scraping`](https://github.com/apify/got-scraping) under the hood.
+Got Scraping is a [Got](https://github.com/sindresorhus/got) extension developed to mimic browser requests, so there's a high chance we'll open the webpage without getting blocked.
+
+## `sendRequest` API
+
+```ts
+async sendRequest(overrideOptions?: GotOptionsInit) => {
+ return gotScraping({
+ url: request.url,
+ method: request.method,
+ body: request.payload,
+ headers: request.headers,
+ proxyUrl: crawlingContext.proxyInfo?.url,
+ sessionToken: session,
+ responseType: 'text',
+ ...overrideOptions,
+ retry: {
+ limit: 0,
+ ...overrideOptions?.retry,
+ },
+ cookieJar: {
+ getCookieString: (url: string) => session!.getCookieString(url),
+ setCookie: (rawCookie: string, url: string) => session!.setCookie(rawCookie, url),
+ ...overrideOptions?.cookieJar,
+ },
+ });
+}
+```
+
+### `url`
+
+By default, it's the URL of current task. However you can override this with a `string` or a `URL` instance if necessary.
+
+*More details in [Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#url).*
+
+### `method`
+
+By default, it's the HTTP method of current task. Possible values are `'GET', 'POST', 'HEAD', 'PUT', 'PATCH', 'DELETE'`.
+
+*More details in [Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#method).*
+
+### `body`
+
+By default, it's the HTTP payload of current task.
+
+*More details in [Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#body).*
+
+### `headers`
+
+By default, it's the HTTP headers of current task. It's an object with `string` values.
+
+*More details in [Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#headers).*
+
+### `proxyUrl`
+
+It's a string representing the proxy server in the format of `protocol://username:password@hostname:port`.
+
+For example, an Apify proxy server looks like this: `http://auto:[email protected]:8000`.
+
+Basic Crawler does not have the concept of a session or proxy, therefore we need to manually pass the `proxyUrl` option:
+
+```ts
+import { BasicCrawler } from 'crawlee';
+
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ const res = await sendRequest({
+ proxyUrl: 'http://auto:[email protected]:8000',
+ });
+ log.info('received body', res.body);
+ },
+});
+```
+
+We use proxies to hide our real IP address.
+
+*More details in [Got Scraping documentation](https://github.com/apify/got-scraping#proxyurl).*
+
+### `sessionToken`
+
+It's a non-primitive object used as a key when generating browser fingerprint. Fingerprints with the same token don't change.
+This can be used to retain the `user-agent` header when using the same Apify Session.
+
+*More details in [Got Scraping documentation](https://github.com/apify/got-scraping#sessiontoken).*
+
+### `responseType`
+
+This option defines how the response should be parsed.
+
+By default, we fetch HTML websites - that is plaintext. Hence, we set `responseType` to `'text'`. However, JSON is possible as well:
+
+```ts
+import { BasicCrawler } from 'crawlee';
+
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ const res = await sendRequest({ responseType: 'json' });
+ log.info('received body', res.body);
+ },
+});
+```
+
+*More details in [Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#responsetype).*
+
+### `cookieJar`
+
+`Got` uses a `cookieJar` to manage cookies. It's an object with an interface of a [`tough-cookie` package](https://github.com/salesforce/tough-cookie).
+
+Example:
+
+```ts
+import { BasicCrawler } from 'crawlee';
+import { CookieJar } from 'tough-cookie';
+
+const cookieJar = new CookieJar();
+
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ const res = await sendRequest({ cookieJar });
+ log.info('received body', res.body);
+ },
+});
+```
+
+*More details in*
+- *[Got documentation](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#cookiejar)*
+- *[Tough Cookie documentation](https://github.com/salesforce/tough-cookie#cookiejarstore-options)*
+
+### `retry.limit`
+
+This option specifies the maximum number of `Got` retries.
+
+By default, `retry.limit` is set to `0`. This is because Crawlee has its own (complicated enough) retry management.
+
+We suggest NOT changing this value for stability reasons.
+
+### `useHeaderGenerator`
+
+It's a boolean for whether to generate browser headers. By default, it's set to `true`, and we recommend keeping this for better results.
+
+### `headerGeneratorOptions`
+
+This option represents an object how to generate browser fingerprint. Example:
+
+```ts
+import { BasicCrawler } from 'crawlee';
+
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ const res = await sendRequest({
+ headerGeneratorOptions: {
+ devices: ['mobile', 'desktop'],
+ locales: ['en-US'],
+ operatingSystems: ['windows', 'macos', 'android', 'ios'],
+ browsers: ['chrome', 'edge', 'firefox', 'safari'],
+ },
+ });
+ log.info('received body', res.body);
+ },
+});
+```
+
+*More details in [`HeaderGeneratorOptions` documentation](https://apify.github.io/fingerprint-suite/api/fingerprint-generator/interface/HeaderGeneratorOptions/).*
+
+**Related links**
+
+- [Got documentation](https://github.com/sindresorhus/got#documentation)
+- [Got Scraping documentation](https://github.com/apify/got-scraping)
+- [Header Generator documentation](https://apify.github.io/fingerprint-suite/docs/guides/fingerprint-generator/)
diff --git a/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright-no-wait.ts b/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright-no-wait.ts
new file mode 100644
index 000000000..77371b51b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright-no-wait.ts
@@ -0,0 +1,14 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ page }) {
+ // Here we don't wait for the selector and immediately
+ // extract the text content from the page.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright.ts b/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright.ts
new file mode 100644
index 000000000..6a1ca9b93
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/javascript-rendering-playwright.ts
@@ -0,0 +1,15 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ page }) {
+ // page.locator points to an element in the DOM
+ // using a CSS selector, but it does not access it yet.
+ const actorCard = page.locator('.ActorStoreItem').first();
+ // Upon calling one of the locator methods Playwright
+ // waits for the element to render and then accesses it.
+ const actorText = await actorCard.textContent();
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer-no-wait.ts b/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer-no-wait.ts
new file mode 100644
index 000000000..c06a4c1b7
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer-no-wait.ts
@@ -0,0 +1,14 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ page }) {
+ // Here we don't wait for the selector and immediately
+ // extract the text content from the page.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer.ts b/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer.ts
new file mode 100644
index 000000000..d4ad7fedf
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/javascript-rendering-puppeteer.ts
@@ -0,0 +1,17 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ page }) {
+ // Puppeteer does not have the automatic waiting functionality
+ // of Playwright, so we have to explicitly wait for the element.
+ await page.waitForSelector('.ActorStoreItem');
+ // Puppeteer does not have helper methods like locator.textContent,
+ // so we have to manually extract the value using in-page JavaScript.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/website/versioned_docs/version-3.7/guides/javascript-rendering.mdx b/website/versioned_docs/version-3.7/guides/javascript-rendering.mdx
new file mode 100644
index 000000000..1fd1b95f7
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/javascript-rendering.mdx
@@ -0,0 +1,94 @@
+---
+id: javascript-rendering
+title: "JavaScript rendering"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import PlaywrightSource from '!!raw-loader!./javascript-rendering-playwright.ts';
+import PlaywrightNoWaitSource from '!!raw-loader!./javascript-rendering-playwright-no-wait.ts';
+import PuppeteerSource from '!!raw-loader!./javascript-rendering-puppeteer.ts';
+import PuppeteerNoWaitSource from '!!raw-loader!./javascript-rendering-puppeteer-no-wait.ts';
+
+JavaScript rendering is the process of executing JavaScript on a page to make changes in the page's structure or content. It's also called client-side rendering, the opposite of server-side rendering. Some modern websites render on the client, some on the server and many cutting edge websites render some things on the server and other things on the client.
+
+The Crawlee website does not use JavaScript rendering to display its content, so we have to look for an example elsewhere. [Apify Store](https://apify.com/store) is a library of scrapers and automations called **actors** that anyone can grab and use for free. It uses JavaScript rendering to display the list of actors, so let's use it to demonstrate how it works.
+
+```ts title="src/main.mjs"
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ $, request }) {
+ // Extract text content of an actor card
+ const actorText = $('.ActorStoreItem').text();
+ console.log(`ACTOR: ${actorText}`);
+ }
+})
+
+await crawler.run(['https://apify.com/store']);
+```
+
+Run the code, and you'll see that the crawler won't print the content of the actor card.
+
+```log
+ACTOR:
+```
+
+That's because Apify Store uses client-side JavaScript to render its content and `CheerioCrawler` can't execute it, so the text never appears in the page's HTML.
+
+You can confirm this using Chrome DevTools. If you go to https://apify.com/store, right-click anywhere in the page, select **View Page Source** and search for **ActorStoreItem** you won't find any results. Then, if you right-click again, select **Inspect** and search for the same **ActorStoreItem**, you will find many of them.
+
+How's this possible? Because **View Page Source** shows the original HTML, before any JavaScript executions. That's what `CheerioCrawler` gets. Whereas with **Inspect** you see the current HTML - after JavaScript execution. When you understand this, it's not a huge surprise that `CheerioCrawler` can't find the data. For that we need a headless browser.
+
+## Headless browsers
+
+To get the contents of `.ActorStoreItem`, you will have to use a headless browser. You can choose from two libraries to control your browser: [Puppeteer](https://github.com/puppeteer/puppeteer) or [Playwright](https://github.com/microsoft/playwright). The choice is simple. If you know one of them, choose the one you know. If you know both, or none, choose Playwright, because it's better in most cases.
+
+## Waiting for elements to render
+
+No matter which library you pick, here's example code for both. Playwright is a little more pleasant to use, but both libraries will get the job done. The big difference between them is that Playwright will automatically wait for elements to appear, whereas in Puppeteer, you have to explicitly wait for them.
+
+<Tabs groupId="javascript-rendering">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PlaywrightSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PuppeteerSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+When you run the code, you'll see the _badly formatted_ content of the first actor card printed to console:
+
+```log
+ACTOR: Web Scraperapify/web-scraperCrawls arbitrary websites using [...]
+```
+
+### We're not kidding
+
+If you don't believe us that the elements need to be waited for, run the following code which skips the waiting.
+
+<Tabs groupId="javascript-rendering">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PlaywrightNoWaitSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PuppeteerNoWaitSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+In both cases, the request will be retried a few times and eventually fail with an error like this:
+
+```log
+ERROR [...] Error: failed to find element matching selector ".ActorStoreItem"
+```
+
+That's because when you try to access the element in the browser, it's not been rendered in the DOM yet.
+
+:::tip
+
+This guide only touches the concept of JavaScript rendering and use of headless browsers. To learn more, continue with the [Puppeteer & Playwright course](https://developers.apify.com/academy/puppeteer-playwright) in the Apify Academy. **It's free and open-source** ❤️.
+
+:::
diff --git a/website/versioned_docs/version-3.7/guides/jsdom_crawler.mdx b/website/versioned_docs/version-3.7/guides/jsdom_crawler.mdx
new file mode 100644
index 000000000..86c402d6d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/jsdom_crawler.mdx
@@ -0,0 +1,61 @@
+---
+id: jsdom-crawler-guide
+title: "JSDOMCrawler guide"
+sidebar_label: "JSDOMCrawler"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+&#8203;<ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> is very useful for scraping with the Window API.
+
+## How the crawler works
+
+&#8203;<ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client. The URLs are fed to the crawler using <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the <ApiLink to="jsdom-crawler/interface/JSDOMCrawlerOptions#additionalMimeTypes">`additionalMimeTypes`</ApiLink> option.
+
+:::info
+
+Modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards, and together they create the final page. To crawl those, see <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.
+
+:::
+
+Once the page's HTML is retrieved, the crawler will pass it to [JSDOM](https://www.npmjs.com/package/jsdom) for parsing. The result is a `window` property, which should be familiar to frontend developers. You can use the Window API to do all sorts of lookups and manipulation of the page's HTML, but in scraping, you will mostly use it to find specific HTML elements and extract their data.
+
+Example use of browser JavaScript:
+
+```ts
+// Return the page title
+document.title; // browsers
+window.document.title; // JSDOM
+```
+
+## When to use `JSDOMCrawler`
+
+`JSDOMCrawler` really shines when `CheerioCrawler` is just not enough. There is an entire set of [APIs](https://developer.mozilla.org/en-US/docs/Web/API/HTML_DOM_API) available!
+
+**Advantages:**
+
+- Easy to set up
+- Familiar for frontend developers
+- Content can be manipulated
+- Automatically avoids some anti-scraping bans
+
+**Disadvantages:**
+
+- Slower than `CheerioCrawler`
+- Does not work for websites that require JavaScript rendering
+- May easily overload the target website with requests
+
+## Example use of Element API
+
+### Find all links on a page
+
+This snippet finds all `<a>` elements which have the `href` attribute and extracts the hrefs into an array.
+
+```js
+Array.from(document.querySelectorAll('a[href]')).map((a) => a.href);
+```
+
+### Other examples
+
+Visit the [Examples](../examples) section to browse examples of `JSDOMCrawler` usage. Almost all examples show `JSDOMCrawler` code in their code tabs.
diff --git a/website/versioned_docs/version-3.7/guides/motivation.mdx b/website/versioned_docs/version-3.7/guides/motivation.mdx
new file mode 100644
index 000000000..e69de29bb
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management.mdx b/website/versioned_docs/version-3.7/guides/proxy_management.mdx
new file mode 100644
index 000000000..50ef9b949
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management.mdx
@@ -0,0 +1,169 @@
+---
+id: proxy-management
+title: Proxy Management
+description: Using proxies to get around those annoying IP-blocks
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import HttpSource from '!!raw-loader!./proxy_management_integration_http.ts';
+import JSDOMSource from '!!raw-loader!./proxy_management_integration_jsdom.ts';
+import CheerioSource from '!!raw-loader!./proxy_management_integration_cheerio.ts';
+import PlaywrightSource from '!!raw-loader!./proxy_management_integration_playwright.ts';
+import PuppeteerSource from '!!raw-loader!./proxy_management_integration_puppeteer.ts';
+import SessionStandaloneSource from '!!raw-loader!./proxy_management_session_standalone.ts';
+import SessionHttpSource from '!!raw-loader!./proxy_management_session_http.ts';
+import SessionJSDOMSource from '!!raw-loader!./proxy_management_session_jsdom.ts';
+import SessionCheerioSource from '!!raw-loader!./proxy_management_session_cheerio.ts';
+import SessionPlaywrightSource from '!!raw-loader!./proxy_management_session_playwright.ts';
+import SessionPuppeteerSource from '!!raw-loader!./proxy_management_session_puppeteer.ts';
+import InspectionHttpSource from '!!raw-loader!./proxy_management_inspection_http.ts';
+import InspectionJSDOMSource from '!!raw-loader!./proxy_management_inspection_jsdom.ts';
+import InspectionCheerioSource from '!!raw-loader!./proxy_management_inspection_cheerio.ts';
+import InspectionPlaywrightSource from '!!raw-loader!./proxy_management_inspection_playwright.ts';
+import InspectionPuppeteerSource from '!!raw-loader!./proxy_management_inspection_puppeteer.ts';
+
+[IP address blocking](https://en.wikipedia.org/wiki/IP_address_blocking) is one of the oldest
+and most effective ways of preventing access to a website. It is therefore paramount for
+a good web scraping library to provide easy to use but powerful tools which can work around
+IP blocking. The most powerful weapon in our anti IP blocking arsenal is a
+[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
+
+With Crawlee we can use our own proxy servers or proxy servers acquired from
+third-party providers.
+
+Check out the [avoid blocking guide](./avoid-blocking) for more information about blocking.
+
+## Quick start
+
+If we already have proxy URLs of our own, we can start using
+them immediately in only a few lines of code.
+
+```javascript
+import { ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ]
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Examples of how to use our proxy URLs with crawlers are shown below in [Crawler integration](#crawler-integration) section.
+
+## Proxy Configuration
+
+All our proxy needs are managed by the <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> class. We create an instance using the `ProxyConfiguration` <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided options. See the <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink> for all the possible constructor options.
+
+### Crawler integration
+
+`ProxyConfiguration` integrates seamlessly into <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink>, <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> and <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.
+
+<Tabs groupId="proxy_session_management">
+ <TabItem value="http" label="HttpCrawler">
+ <CodeBlock language="js">
+ {HttpSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="cheerio" label="CheerioCrawler" default>
+ <CodeBlock language="js">
+ {CheerioSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="jsdom" label="JSDOMCrawler">
+ <CodeBlock language="js">
+ {JSDOMSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js">
+ {PlaywrightSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {PuppeteerSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Our crawlers will now use the selected proxies for all connections.
+
+### IP Rotation and session management
+
+&#8203;<ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> allows us to pass a `sessionId` parameter. It will then be used to create a `sessionId`-`proxyUrl` pair, and subsequent `newUrl()` calls with the same `sessionId` will always return the same `proxyUrl`. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
+
+When no `sessionId` is provided, our proxy URLs are rotated round-robin.
+
+<Tabs groupId="proxy_session_management">
+ <TabItem value="http" label="HttpCrawler">
+ <CodeBlock language="js">
+ {SessionHttpSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="cheerio" label="CheerioCrawler" default>
+ <CodeBlock language="js">
+ {SessionCheerioSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="jsdom" label="JSDOMCrawler">
+ <CodeBlock language="js">
+ {SessionJSDOMSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js">
+ {SessionPlaywrightSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {SessionPuppeteerSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="standalone" label="Standalone">
+ <CodeBlock language="js">
+ {SessionStandaloneSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+## Inspecting current proxy in Crawlers
+
+`HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
+in their `requestHandler` using a <ApiLink to="core/interface/ProxyInfo">`proxyInfo`</ApiLink> object.
+With the `proxyInfo` object, we can easily access the proxy URL.
+
+<Tabs groupId="proxy_session_management">
+ <TabItem value="http" label="HttpCrawler">
+ <CodeBlock language="js">
+ {InspectionHttpSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="cheerio" label="CheerioCrawler" default>
+ <CodeBlock language="js">
+ {InspectionCheerioSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="jsdom" label="JSDOMCrawler">
+ <CodeBlock language="js">
+ {InspectionJSDOMSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js">
+ {InspectionPlaywrightSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {InspectionPuppeteerSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_inspection_cheerio.ts b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_cheerio.ts
new file mode 100644
index 000000000..68bdd2125
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_cheerio.ts
@@ -0,0 +1,11 @@
+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new CheerioCrawler({
+ proxyConfiguration,
+ async requestHandler({ proxyInfo }) {
+ console.log(proxyInfo);
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_inspection_http.ts b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_http.ts
new file mode 100644
index 000000000..076350877
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_http.ts
@@ -0,0 +1,11 @@
+import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new HttpCrawler({
+ proxyConfiguration,
+ async requestHandler({ proxyInfo }) {
+ console.log(proxyInfo);
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_inspection_jsdom.ts b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_jsdom.ts
new file mode 100644
index 000000000..db7607143
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_jsdom.ts
@@ -0,0 +1,11 @@
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new JSDOMCrawler({
+ proxyConfiguration,
+ async requestHandler({ proxyInfo }) {
+ console.log(proxyInfo);
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_inspection_playwright.ts b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_playwright.ts
new file mode 100644
index 000000000..db4b2d277
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_playwright.ts
@@ -0,0 +1,11 @@
+import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PlaywrightCrawler({
+ proxyConfiguration,
+ async requestHandler({ proxyInfo }) {
+ console.log(proxyInfo);
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_inspection_puppeteer.ts b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_puppeteer.ts
new file mode 100644
index 000000000..154ce0068
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_inspection_puppeteer.ts
@@ -0,0 +1,11 @@
+import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PuppeteerCrawler({
+ proxyConfiguration,
+ async requestHandler({ proxyInfo }) {
+ console.log(proxyInfo);
+ },
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_integration_cheerio.ts b/website/versioned_docs/version-3.7/guides/proxy_management_integration_cheerio.ts
new file mode 100644
index 000000000..74f9878c0
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_integration_cheerio.ts
@@ -0,0 +1,13 @@
+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ],
+});
+
+const crawler = new CheerioCrawler({
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_integration_http.ts b/website/versioned_docs/version-3.7/guides/proxy_management_integration_http.ts
new file mode 100644
index 000000000..0db0a9ab9
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_integration_http.ts
@@ -0,0 +1,13 @@
+import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ],
+});
+
+const crawler = new HttpCrawler({
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_integration_jsdom.ts b/website/versioned_docs/version-3.7/guides/proxy_management_integration_jsdom.ts
new file mode 100644
index 000000000..4164ef8cf
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_integration_jsdom.ts
@@ -0,0 +1,13 @@
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ],
+});
+
+const crawler = new JSDOMCrawler({
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_integration_playwright.ts b/website/versioned_docs/version-3.7/guides/proxy_management_integration_playwright.ts
new file mode 100644
index 000000000..ad41b3775
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_integration_playwright.ts
@@ -0,0 +1,13 @@
+import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ],
+});
+
+const crawler = new PlaywrightCrawler({
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_integration_puppeteer.ts b/website/versioned_docs/version-3.7/guides/proxy_management_integration_puppeteer.ts
new file mode 100644
index 000000000..f4dc43ac3
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_integration_puppeteer.ts
@@ -0,0 +1,13 @@
+import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({
+ proxyUrls: [
+ 'http://proxy-1.com',
+ 'http://proxy-2.com',
+ ],
+});
+
+const crawler = new PuppeteerCrawler({
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_cheerio.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_cheerio.ts
new file mode 100644
index 000000000..f8484e6ce
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_cheerio.ts
@@ -0,0 +1,10 @@
+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new CheerioCrawler({
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_http.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_http.ts
new file mode 100644
index 000000000..601e26812
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_http.ts
@@ -0,0 +1,10 @@
+import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new HttpCrawler({
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_jsdom.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_jsdom.ts
new file mode 100644
index 000000000..2321f97de
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_jsdom.ts
@@ -0,0 +1,10 @@
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new JSDOMCrawler({
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_playwright.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_playwright.ts
new file mode 100644
index 000000000..220868469
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_playwright.ts
@@ -0,0 +1,10 @@
+import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PlaywrightCrawler({
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_puppeteer.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_puppeteer.ts
new file mode 100644
index 000000000..5bcbb2806
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_puppeteer.ts
@@ -0,0 +1,10 @@
+import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PuppeteerCrawler({
+ useSessionPool: true,
+ persistCookiesPerSession: true,
+ proxyConfiguration,
+ // ...
+});
diff --git a/website/versioned_docs/version-3.7/guides/proxy_management_session_standalone.ts b/website/versioned_docs/version-3.7/guides/proxy_management_session_standalone.ts
new file mode 100644
index 000000000..3723d5072
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/proxy_management_session_standalone.ts
@@ -0,0 +1,9 @@
+import { ProxyConfiguration, SessionPool } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const sessionPool = await SessionPool.open({ /* opts */ });
+
+const session = await sessionPool.getSession();
+
+const proxyUrl = await proxyConfiguration.newUrl(session.id);
diff --git a/website/versioned_docs/version-3.7/guides/request_storage.mdx b/website/versioned_docs/version-3.7/guides/request_storage.mdx
new file mode 100644
index 000000000..2993b8371
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage.mdx
@@ -0,0 +1,146 @@
+---
+id: request-storage
+title: Request Storage
+description: How to store the requests your crawler will go through
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import BasicOperationsSource from '!!raw-loader!./request_storage_queue_basic.ts';
+import CrawlerExplicitSource from '!!raw-loader!./request_storage_queue_crawler_explicit.ts';
+import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';
+
+import RequestQueueListSource from '!!raw-loader!./request_storage_queue_list.ts';
+import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';
+
+Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
+
+## Request queue
+
+The request queue is a storage of URLs to crawl. The queue is used for the deep crawling of websites, where we start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
+
+Each Crawlee project run is associated with a **default request queue**. Typically, it is used to store URLs to crawl in the specific crawler run. Its usage is optional.
+
+In Crawlee, the request queue is represented by the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> class.
+
+The request queue is managed by <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink> class and its data is stored in memory, while also being off-loaded to the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+
+```
+{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
+```
+
+:::note
+
+`{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_REQUEST_QUEUE_ID` environment variable.
+
+:::
+
+:::note
+
+`entries.json` contains an array or requests.
+
+:::
+
+The following code demonstrates the usage of the request queue:
+
+<Tabs groupId="request_queue">
+ <TabItem value="crawler" label="Usage with Crawler" default>
+ <CodeBlock language="js">
+ {CrawlerSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="crawler_explicit" label="Explicit usage with Crawler">
+ <CodeBlock language="js">
+ {CrawlerExplicitSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="basic_operations" label="Basic Operations" default>
+ <CodeBlock language="js">
+ {BasicOperationsSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.
+
+## Request list
+
+The request list is not a storage per se - it represents the list of URLs to crawl that is stored in a crawler run memory (or optionally in default [Key-Value Store](../guides/result-storage#key-value-store) associated with the run, if specified). The list is used for the crawling of a large number of URLs, when we know all the URLs which should be visited by the crawler and no URLs would be added during the run. The URLs can be provided either in code or parsed from a text file hosted on the web.
+
+Request list is created exclusively for the crawler run and only if its usage is explicitly specified in the code. Its usage is optional.
+
+In Crawlee, the request list is represented by the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> class.
+
+The following code demonstrates basic operations of the request list:
+
+```javascript
+import { RequestList, PuppeteerCrawler } from 'crawlee';
+
+// Prepare the sources array with URLs to visit
+const sources = [
+ { url: 'http://www.example.com/page-1' },
+ { url: 'http://www.example.com/page-2' },
+ { url: 'http://www.example.com/page-3' },
+];
+
+// Open the request list.
+// List name is used to persist the sources and the list state in the key-value store
+const requestList = await RequestList.open('my-list', sources);
+
+// The crawler will automatically process requests from the list
+// It's used the same way for Cheerio /Playwright crawlers.
+const crawler = new PuppeteerCrawler({
+ requestList,
+ async requestHandler({ page, request }) {
+ // Process the page (extract data, take page screenshot, etc).
+ // No more requests could be added to the request list here
+ },
+});
+```
+
+## Which one to choose?
+
+When using Request queue - we would normally have several start URLs (e.g. category pages on e-commerce website) and then recursively add more (e.g. individual item pages) programmatically to the queue, it supports dynamic adding and removing of requests. No more URLs can be added to Request list after its initialization as it is immutable, URLs cannot be removed from the list either.
+
+On the other hand, the Request queue is not optimized for adding or removing numerous URLs in a batch. This is technically possible, but requests are added one by one to the queue, and thus it would take significant time with a larger number of requests. Request list however can contain even millions of URLs, and it would take significantly less time to add them to the list, compared to the queue.
+
+Note that Request queue and Request list can be used together by the same crawler. In such cases, each request from the Request list is enqueued into the Request queue first (to the foremost position in the queue, even if Request queue is not empty) and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue). In practical terms, such a combination can be useful when there are numerous initial URLs, but more URLs would be added dynamically by the crawler.
+
+:::tip
+
+In Crawlee, there is no much need to combine the request queue together with the request list (although it's technically possible).
+
+Previously there was no way to add the initial requests to the queue in batches (to add an array of requests), i.e. we could have only added the requests one by one to the queue with the help of <ApiLink to="core/class/RequestQueue#addRequest">`addRequest()`</ApiLink> function.
+
+However, now we could use the <ApiLink to="core/class/RequestQueue#addRequests">`addRequests()`</ApiLink> function, which adds requests in batches. Thus, instead of combining the request queue and the request list, we can use only the request queue for such use-cases now. See the examples below.
+
+:::
+
+<Tabs groupId="queue_list">
+ <TabItem value="add_requests" label="Request Queue" default>
+ <CodeBlock language="js">
+ {RequestQueueAddRequestsSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="queue_list" label="Request Queue + Request List">
+ <CodeBlock language="js">
+ {RequestQueueListSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+## Cleaning up the storages
+
+Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the <ApiLink to="core/function/purgeDefaultStorages">`purgeDefaultStorages()`</ApiLink> helper explicitly:
+
+```javascript
+import { purgeDefaultStorages } from 'crawlee';
+
+await purgeDefaultStorages();
+```
+
+Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/website/versioned_docs/version-3.7/guides/request_storage_queue_basic.ts b/website/versioned_docs/version-3.7/guides/request_storage_queue_basic.ts
new file mode 100644
index 000000000..66d3d3372
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage_queue_basic.ts
@@ -0,0 +1,17 @@
+import { RequestQueue } from 'crawlee';
+
+// Open the default request queue associated with the crawler run
+const requestQueue = await RequestQueue.open();
+
+// Enqueue the initial batch of requests (could be an array of just one)
+await requestQueue.addRequests([
+ { url: 'https://example.com/1' },
+ { url: 'https://example.com/2' },
+ { url: 'https://example.com/3' },
+]);
+
+// Open the named request queue
+const namedRequestQueue = await RequestQueue.open('named-queue');
+
+// Remove the named request queue
+await namedRequestQueue.drop();
diff --git a/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler.ts b/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler.ts
new file mode 100644
index 000000000..4815ea71a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler.ts
@@ -0,0 +1,25 @@
+import { CheerioCrawler } from 'crawlee';
+
+// The crawler will automatically process requests from the queue.
+// It's used the same way for Puppeteer/Playwright crawlers.
+const crawler = new CheerioCrawler({
+ // Note that we're not specifying the requestQueue here
+ async requestHandler({ $, crawler, enqueueLinks }) {
+ // Add new request to the queue
+ await crawler.addRequests([{ url: 'https://example.com/new-page' }]);
+ // Add links found on page to the queue
+ await enqueueLinks();
+ },
+});
+
+// Add the initial requests.
+// Note that we are not opening the request queue explicitly before
+await crawler.addRequests([
+ { url: 'https://example.com/1' },
+ { url: 'https://example.com/2' },
+ { url: 'https://example.com/3' },
+ // ...
+]);
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler_explicit.ts b/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler_explicit.ts
new file mode 100644
index 000000000..50bff9757
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage_queue_crawler_explicit.ts
@@ -0,0 +1,27 @@
+import { RequestQueue, CheerioCrawler } from 'crawlee';
+
+// Open the default request queue associated with the current run
+const requestQueue = await RequestQueue.open();
+
+// Enqueue the initial requests
+await requestQueue.addRequests([
+ { url: 'https://example.com/1' },
+ { url: 'https://example.com/2' },
+ { url: 'https://example.com/3' },
+ // ...
+]);
+
+// The crawler will automatically process requests from the queue.
+// It's used the same way for Puppeteer/Playwright crawlers
+const crawler = new CheerioCrawler({
+ requestQueue,
+ async requestHandler({ $, request, enqueueLinks }) {
+ // Add new request to the queue
+ await requestQueue.addRequests([{ url: 'https://example.com/new-page' }]);
+ // Add links found on page to the queue
+ await enqueueLinks();
+ },
+});
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/guides/request_storage_queue_list.ts b/website/versioned_docs/version-3.7/guides/request_storage_queue_list.ts
new file mode 100644
index 000000000..456caf1c0
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage_queue_list.ts
@@ -0,0 +1,42 @@
+// This is technically correct, but
+// we need to explicitly open/use both the request queue and the request list.
+// We suggest using the request queue and batch add the requests instead.
+
+import { RequestList, RequestQueue, PuppeteerCrawler } from 'crawlee';
+
+// Prepare the sources array with URLs to visit (it can contain millions of URLs)
+const sources = [
+ { url: 'http://www.example.com/page-1' },
+ { url: 'http://www.example.com/page-2' },
+ { url: 'http://www.example.com/page-3' },
+ // ...
+];
+
+// Open the request list with the initial sources array
+const requestList = await RequestList.open('my-list', sources);
+
+// Open the default request queue. It's not necessary to add any requests to the queue
+const requestQueue = await RequestQueue.open();
+
+// The crawler will automatically process requests from the list and the queue.
+// It's used the same way for Cheerio/Playwright crawlers
+const crawler = new PuppeteerCrawler({
+ requestList,
+ requestQueue,
+ // Each request from the request list is enqueued to the request queue one by one.
+ // At this point request with the same URL would exist in the list and the queue
+ async requestHandler({ crawler, enqueueLinks }) {
+ // Add new request to the queue
+ await crawler.addRequests(['http://www.example.com/new-page']);
+
+ // Add links found on page to the queue
+ await enqueueLinks();
+
+ // The requests above would be added to the queue (but not to the list)
+ // and would be processed after the request list is empty.
+ // No more requests could be added to the list here
+ },
+});
+
+// Run the crawler
+await crawler.run();
diff --git a/website/versioned_docs/version-3.7/guides/request_storage_queue_only.ts b/website/versioned_docs/version-3.7/guides/request_storage_queue_only.ts
new file mode 100644
index 000000000..5d9a31379
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/request_storage_queue_only.ts
@@ -0,0 +1,32 @@
+// This is the suggested way.
+// Note that we are not using the request list at all,
+// and not using the request queue explicitly here.
+
+import { PuppeteerCrawler } from 'crawlee';
+
+// Prepare the sources array with URLs to visit (it can contain millions of URLs)
+const sources = [
+ { url: 'http://www.example.com/page-1' },
+ { url: 'http://www.example.com/page-2' },
+ { url: 'http://www.example.com/page-3' },
+ // ...
+];
+
+// The crawler will automatically process requests from the queue.
+// It's used the same way for Cheerio/Playwright crawlers
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ crawler, enqueueLinks }) {
+ // Add new request to the queue
+ await crawler.addRequests(['http://www.example.com/new-page']);
+
+ // Add links found on page to the queue
+ await enqueueLinks();
+
+ // The requests above would be added to the queue
+ // and would be processed after the initial requests are processed.
+ },
+});
+
+// Add the initial sources array to the request queue
+// and run the crawler
+await crawler.run(sources);
diff --git a/website/versioned_docs/version-3.7/guides/result_storage.mdx b/website/versioned_docs/version-3.7/guides/result_storage.mdx
new file mode 100644
index 000000000..eff313aef
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/result_storage.mdx
@@ -0,0 +1,113 @@
+---
+id: result-storage
+title: Result Storage
+description: Where are you going to store all of that juicy scraped data?!
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
+
+Crawlee storage is managed by <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink> class. During the crawler run all information is stored in memory, while also being off-loaded to the local files in respective storage type folders.
+
+## Key-value store
+
+The key-value store is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.
+
+Each Crawlee project run is associated with a **default key-value store**. By convention, the project input and output are stored in the default key-value store under the `INPUT` and `OUTPUT` keys respectively. Typically, both input and output are JSON files, although they could be any other format.
+
+In Crawlee, the key-value store is represented by the <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink> class. In order to simplify access to the default key-value store, Crawlee also provides <ApiLink to="core/class/KeyValueStore#getValue">`KeyValueStore.getValue()`</ApiLink> and <ApiLink to="core/class/KeyValueStore#setValue">`KeyValueStore.setValue()`</ApiLink> functions.
+
+The data is stored in the directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+
+```
+{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
+```
+
+:::note
+
+`{STORE_ID}` is the name or the ID of the key-value store. The default key-value store has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.
+
+:::
+
+The following code demonstrates basic operations of key-value stores:
+
+```javascript
+import { KeyValueStore } from 'crawlee';
+
+// Get the INPUT from the default key-value store
+const input = await KeyValueStore.getInput();
+
+// Write the OUTPUT to the default key-value store
+await KeyValueStore.setValue('OUTPUT', { myResult: 123 });
+
+// Open a named key-value store
+const store = await KeyValueStore.open('some-name');
+
+// Write a record to the named key-value store.
+// JavaScript object is automatically converted to JSON,
+// strings and binary buffers are stored as they are
+await store.setValue('some-key', { foo: 'bar' });
+
+// Read a record from the named key-value store.
+// Note that JSON is automatically parsed to a JavaScript object,
+// text data is returned as a string, and other data is returned as binary buffer
+const value = await store.getValue('some-key');
+
+// Delete a record from the named key-value store
+await store.setValue('some-key', null);
+```
+
+To see a real-world example of how to get the input from the key-value store, see the [Screenshots](../examples/capture-screenshot) example.
+
+## Dataset
+
+Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. Dataset can be imagined as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - we can only add new records to it, but we cannot modify or remove existing records.
+
+Each Crawlee project run is associated with a **default dataset**. Typically, it is used to store crawling results specific for the crawler run. Its usage is optional.
+
+In Crawlee, the dataset is represented by the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> class. In order to simplify writes to the default dataset, Crawlee also provides the <ApiLink to="core/class/Dataset#pushData">`Dataset.pushData()`</ApiLink> function.
+
+The data is stored in the directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+
+```
+{CRAWLEE_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
+```
+
+:::note
+
+`{DATASET_ID}` is the name or the ID of the dataset. The default dataset has ID `default`, unless we override it by setting the `CRAWLEE_DEFAULT_DATASET_ID` environment variable. Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.
+
+:::
+
+The following code demonstrates basic operations of the dataset:
+
+```javascript
+import { Dataset } from 'crawlee';
+
+// Write a single row to the default dataset
+await Dataset.pushData({ col1: 123, col2: 'val2' });
+
+// Open a named dataset
+const dataset = await Dataset.open('some-name');
+
+// Write a single row
+await dataset.pushData({ foo: 'bar' });
+
+// Write multiple rows
+await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);
+```
+
+To see how to use the dataset to store crawler results, see the [Cheerio Crawler](../examples/cheerio-crawler) example.
+
+## Cleaning up the storages
+
+Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the <ApiLink to="core/function/purgeDefaultStorages">`purgeDefaultStorages()`</ApiLink> helper explicitly:
+
+```javascript
+import { purgeDefaultStorages } from 'crawlee';
+
+await purgeDefaultStorages();
+```
+
+Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the <ApiLink to="core/interface/StorageClient">`StorageClient`</ApiLink> interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
diff --git a/website/versioned_docs/version-3.7/guides/scaling_crawlers.mdx b/website/versioned_docs/version-3.7/guides/scaling_crawlers.mdx
new file mode 100644
index 000000000..e296cc909
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/scaling_crawlers.mdx
@@ -0,0 +1,120 @@
+---
+id: scaling-crawlers
+title: Scaling our crawlers
+description: To infinity and beyond! ...within limits
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import CodeBlock from '@theme/CodeBlock';
+
+import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';
+import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';
+import AutoscaledPoolOptionsSource from '!!raw-loader!./scaling_crawlers_autoscaledPoolOptions.ts';
+
+As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.
+
+:::tip
+
+All of these options are available on all crawlers Crawlee provides, but for this guide we'll be using the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>. We can see all options that are available <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions">`here`</ApiLink>.
+:::
+
+## `maxRequestsPerMinute`
+
+This controls how many total requests can be made per minute. It counts the amount of requests done every second, to ensure there is not a burst of requests at the `maxConcurrency` limit followed by a long period of waiting. By default, it is set to `Infinity` which means the crawler will keep going up to the `maxConcurrency`. We would set this if we wanted our crawler to work at full throughput, but also not keep hitting the website we're crawling with non-stop requests.
+
+<CodeBlock language="js">
+ {MaxRequestsPerMinuteSource}
+</CodeBlock>
+
+## `minConcurrency` and `maxConcurrency`
+
+These control how many parallel requests can be run at any time. By default, crawlers will start with one parallel request at a time and scale up over time to a maximum of 200 requests at a time.
+
+:::caution Don't set `minConcurrency` too high!
+
+Setting this option too high compared to the available system resources will make your crawler run extremely slow or might even crash.
+
+It's recommended to leave it at the default value that is provided and letting the crawler scale up and down automatically based on available resources instead.
+
+:::
+
+<CodeBlock language="js">
+ {MinMaxConcurrencySource}
+</CodeBlock>
+
+## Advanced options
+
+While the options above should be enough for most users, if we wanted to get super deep into the configuration of the autoscaling pool (the internal utility in Crawlee that helps us allow crawlers to scale up and down), we can do so through the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#autoscaledPoolOptions">`autoscaledPoolOptions`</ApiLink> object available on crawler options.
+
+:::danger Complex options up ahead!
+
+This section is super advanced and, unless you test the changes extensively and know what you're doing, it's better to leave these options to their defaults, as they are most likely going to work fine without much fuss.
+
+:::
+
+With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler:
+
+<CodeBlock language="js">
+ {AutoscaledPoolOptionsSource}
+</CodeBlock>
+
+### `desiredConcurrency`
+
+This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
+
+### `desiredConcurrencyRatio`
+
+The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.95`.
+
+We can think of this as the point where the autoscaling pool can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
+
+### `scaleUpStepRatio` and `scaleDownStepRatio`
+
+These values define the fractional amount of desired concurrency to be added or subtracted as the autoscaling pool scales up or down. Both of these values default to `0.05`.
+
+Every time the autoscaled pool attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
+
+### `maybeRunIntervalSecs`
+
+Indicates how often the autoscaling pool should check if more requests can be started and, if that's true, starts a new request if there are any available. This value is represented in seconds, and defaults to `0.5`.
+
+:::info
+
+Changing this has no effect for requests that are fired immediately after the previous ones are finished. However, it will influence how fast new requests will be started after the autoscaled pool scales up.
+
+:::
+
+### `loggingIntervalSecs`
+
+This option lets us control how often the autoscaled pool should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
+
+We can disable logging altogether by setting this to `null`. By default, it is set to `60` seconds.
+
+### `autoscaleIntervalSecs`
+
+This option lets us control how often the autoscaling pool should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
+
+:::tip
+
+It's recommended you keep this value between `5` and `20` seconds.
+
+:::
+
+:::caution Be careful with how low, or high, you set this option
+
+Setting this option to a value that's too low might have a severe impact on our crawling performance. And, in reverse, setting this to a value that's too high might mean we leave performance on the table that could've been used for crawling more requests instead.
+
+With that said, if you configure this alongside [`scaleUpStepRatio` and `scaleDownStepRatio`](#scaleupstepratio-and-scaledownstepratio), you could make your crawler scale up at a slower interval, but with more requests at a time when it does.
+
+:::
+
+### `maxTasksPerMinute`
+
+This controls how many total requests can be made per minute. It counts the amount of requests done every second, to ensure there is not a burst of requests at the `maxConcurrency` limit followed by a long period of waiting. By default, it is set to `Infinity` which means the crawler will keep going up to the `maxConcurrency`. We would set this if we wanted our crawler to work at full throughput, but also not keep hitting the website we're crawl with non-stop requests.
+
+:::info
+
+This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestsperminute) in your crawler options too, as it is a shortcut for visibility and ease of access.
+
+:::
diff --git a/website/versioned_docs/version-3.7/guides/scaling_crawlers_autoscaledPoolOptions.ts b/website/versioned_docs/version-3.7/guides/scaling_crawlers_autoscaledPoolOptions.ts
new file mode 100644
index 000000000..5d8f53c4d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/scaling_crawlers_autoscaledPoolOptions.ts
@@ -0,0 +1,8 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Pass in advanced options by providing them in the autoscaledPoolOptions
+ autoscaledPoolOptions: {
+ // ...
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/scaling_crawlers_maxRequestsPerMinute.ts b/website/versioned_docs/version-3.7/guides/scaling_crawlers_maxRequestsPerMinute.ts
new file mode 100644
index 000000000..43ae565b7
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/scaling_crawlers_maxRequestsPerMinute.ts
@@ -0,0 +1,8 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Let the crawler know it can run up to 100 requests concurrently at any time
+ maxConcurrency: 100,
+ // ...but also ensure the crawler never exceeds 250 requests per minute
+ maxRequestsPerMinute: 250,
+});
diff --git a/website/versioned_docs/version-3.7/guides/scaling_crawlers_minMaxConcurrency.ts b/website/versioned_docs/version-3.7/guides/scaling_crawlers_minMaxConcurrency.ts
new file mode 100644
index 000000000..06a562a4b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/scaling_crawlers_minMaxConcurrency.ts
@@ -0,0 +1,8 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Start the crawler right away and ensure there will always be 5 concurrent requests ran at any time
+ minConcurrency: 5,
+ // Ensure the crawler doesn't exceed 15 concurrent requests ran at any time
+ maxConcurrency: 15,
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management.mdx b/website/versioned_docs/version-3.7/guides/session_management.mdx
new file mode 100644
index 000000000..ad0eca80e
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management.mdx
@@ -0,0 +1,81 @@
+---
+id: session-management
+title: Session Management
+description: How to manage your cookies, proxy IP rotations and more
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import BasicSource from '!!raw-loader!./session_management_basic.ts';
+import HttpSource from '!!raw-loader!./session_management_http.ts';
+import CheerioSource from '!!raw-loader!./session_management_cheerio.ts';
+import JSDOMSource from '!!raw-loader!./session_management_jsdom.ts';
+import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';
+import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';
+import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';
+
+&#8203;<ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> is a class that allows us to handle the rotation of proxy IP addresses along with cookies and other custom settings in Crawlee.
+
+The main benefit of using Session pool is that we can filter out blocked or non-working proxies,
+so our actor does not retry requests over known blocked/non-working proxies.
+Another benefit of using SessionPool is that we can store information tied tightly to an IP address,
+such as cookies, auth tokens, and particular headers. Having our cookies and other identifiers used only with a specific IP will reduce the chance of being blocked.
+The last but not least benefit is the even rotation of IP addresses - SessionPool picks the session randomly,
+which should prevent burning out a small pool of available IPs.
+
+Check out the [avoid blocking guide](./avoid-blocking) for more information about blocking.
+
+Now let's take a look at the examples of how to use Session pool:
+- with <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>;
+- with <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink>;
+- with <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>;
+- with <ApiLink to="jsdom-crawler/class/JSDOMCrawler">`JSDOMCrawler`</ApiLink>;
+- with <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>;
+- with <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>;
+- without a crawler (standalone usage to manage sessions manually).
+
+<Tabs groupId="session_pool">
+ <TabItem value="basic" label="BasicCrawler">
+ <CodeBlock language="js">
+ {BasicSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="http" label="HttpCrawler">
+ <CodeBlock language="js">
+ {HttpSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="cheerio" label="CheerioCrawler" default>
+ <CodeBlock language="js">
+ {CheerioSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="jsdom" label="JSDOMCrawler">
+ <CodeBlock language="js">
+ {JSDOMSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js">
+ {PlaywrightSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js">
+ {PuppeteerSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="standalone" label="Standalone">
+ <CodeBlock language="js">
+ {StandaloneSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+These are the basics of configuring SessionPool.
+Please, bear in mind that a Session pool needs time to find working IPs and build up the pool,
+so we will probably see a lot of errors until it becomes stabilized.
diff --git a/website/versioned_docs/version-3.7/guides/session_management_basic.ts b/website/versioned_docs/version-3.7/guides/session_management_basic.ts
new file mode 100644
index 000000000..8e7d41cca
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_basic.ts
@@ -0,0 +1,54 @@
+import { BasicCrawler, ProxyConfiguration } from 'crawlee';
+import { gotScraping } from 'got-scraping';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new BasicCrawler({
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration.
+ sessionPoolOptions: { maxPoolSize: 100 },
+ async requestHandler({ request, session }) {
+ const { url } = request;
+ const requestOptions = {
+ url,
+ // We use session id in order to have the same proxyUrl
+ // for all the requests using the same session.
+ proxyUrl: await proxyConfiguration.newUrl(session.id),
+ throwHttpErrors: false,
+ headers: {
+ // If you want to use the cookieJar.
+ // This way you get the Cookie headers string from session.
+ Cookie: session.getCookieString(url),
+ },
+ };
+ let response;
+
+ try {
+ response = await gotScraping(requestOptions);
+ } catch (e) {
+ if (e === 'SomeNetworkError') {
+ // If a network error happens, such as timeout, socket hangup, etc.
+ // There is usually a chance that it was just bad luck
+ // and the proxy works. No need to throw it away.
+ session.markBad();
+ }
+ throw e;
+ }
+
+ // Automatically retires the session based on response HTTP status code.
+ session.retireOnBlockedStatusCodes(response.statusCode);
+
+ if (response.body.blocked) {
+ // You are sure it is blocked.
+ // This will throw away the session.
+ session.retire();
+ }
+
+ // Everything is ok, you can get the data.
+ // No need to call session.markGood -> BasicCrawler calls it for you.
+
+ // If you want to use the CookieJar in session you need.
+ session.setCookiesFromResponse(response);
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_cheerio.ts b/website/versioned_docs/version-3.7/guides/session_management_cheerio.ts
new file mode 100644
index 000000000..397256b95
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_cheerio.ts
@@ -0,0 +1,26 @@
+import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new CheerioCrawler({
+ // To use the proxy IP session rotation logic, you must turn the proxy usage on.
+ proxyConfiguration,
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration.
+ sessionPoolOptions: { maxPoolSize: 100 },
+ // Set to true if you want the crawler to save cookies per session,
+ // and set the cookie header to request automatically (default is true).
+ persistCookiesPerSession: true,
+ async requestHandler({ session, $ }) {
+ const title = $('title').text();
+
+ if (title === 'Blocked') {
+ session.retire();
+ } else if (title === 'Not sure if blocked, might also be a connection error') {
+ session.markBad();
+ } else {
+ // session.markGood() - this step is done automatically in BasicCrawler.
+ }
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_http.ts b/website/versioned_docs/version-3.7/guides/session_management_http.ts
new file mode 100644
index 000000000..b6d40d3ec
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_http.ts
@@ -0,0 +1,26 @@
+import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new HttpCrawler({
+ // To use the proxy IP session rotation logic, you must turn the proxy usage on.
+ proxyConfiguration,
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration.
+ sessionPoolOptions: { maxPoolSize: 100 },
+ // Set to true if you want the crawler to save cookies per session,
+ // and set the cookie header to request automatically (default is true).
+ persistCookiesPerSession: true,
+ async requestHandler({ session, body }) {
+ const title = body.match(/<title(?:.*?)>(.*?)<\/title>/)?.[1];
+
+ if (title === 'Blocked') {
+ session.retire();
+ } else if (title === 'Not sure if blocked, might also be a connection error') {
+ session.markBad();
+ } else {
+ // session.markGood() - this step is done automatically in BasicCrawler.
+ }
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_jsdom.ts b/website/versioned_docs/version-3.7/guides/session_management_jsdom.ts
new file mode 100644
index 000000000..6f57ca9ba
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_jsdom.ts
@@ -0,0 +1,26 @@
+import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new JSDOMCrawler({
+ // To use the proxy IP session rotation logic, you must turn the proxy usage on.
+ proxyConfiguration,
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration.
+ sessionPoolOptions: { maxPoolSize: 100 },
+ // Set to true if you want the crawler to save cookies per session,
+ // and set the cookie header to request automatically (default is true).
+ persistCookiesPerSession: true,
+ async requestHandler({ session, window }) {
+ const title = window.document.title;
+
+ if (title === 'Blocked') {
+ session.retire();
+ } else if (title === 'Not sure if blocked, might also be a connection error') {
+ session.markBad();
+ } else {
+ // session.markGood() - this step is done automatically in BasicCrawler.
+ }
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_playwright.ts b/website/versioned_docs/version-3.7/guides/session_management_playwright.ts
new file mode 100644
index 000000000..f4ea5651d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_playwright.ts
@@ -0,0 +1,26 @@
+import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PlaywrightCrawler({
+ // To use the proxy IP session rotation logic, you must turn the proxy usage on.
+ proxyConfiguration,
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration
+ sessionPoolOptions: { maxPoolSize: 100 },
+ // Set to true if you want the crawler to save cookies per session,
+ // and set the cookies to page before navigation automatically (default is true).
+ persistCookiesPerSession: true,
+ async requestHandler({ page, session }) {
+ const title = await page.title();
+
+ if (title === 'Blocked') {
+ session.retire();
+ } else if (title === 'Not sure if blocked, might also be a connection error') {
+ session.markBad();
+ } else {
+ // session.markGood() - this step is done automatically in PlaywrightCrawler.
+ }
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_puppeteer.ts b/website/versioned_docs/version-3.7/guides/session_management_puppeteer.ts
new file mode 100644
index 000000000..8f7aad903
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_puppeteer.ts
@@ -0,0 +1,26 @@
+import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+
+const proxyConfiguration = new ProxyConfiguration({ /* opts */ });
+
+const crawler = new PuppeteerCrawler({
+ // To use the proxy IP session rotation logic, you must turn the proxy usage on.
+ proxyConfiguration,
+ // Activates the Session pool (default is true).
+ useSessionPool: true,
+ // Overrides default Session pool configuration
+ sessionPoolOptions: { maxPoolSize: 100 },
+ // Set to true if you want the crawler to save cookies per session,
+ // and set the cookies to page before navigation automatically (default is true).
+ persistCookiesPerSession: true,
+ async requestHandler({ page, session }) {
+ const title = await page.title();
+
+ if (title === 'Blocked') {
+ session.retire();
+ } else if (title === 'Not sure if blocked, might also be a connection error') {
+ session.markBad();
+ } else {
+ // session.markGood() - this step is done automatically in PuppeteerCrawler.
+ }
+ },
+});
diff --git a/website/versioned_docs/version-3.7/guides/session_management_standalone.ts b/website/versioned_docs/version-3.7/guides/session_management_standalone.ts
new file mode 100644
index 000000000..c6fa33d82
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/session_management_standalone.ts
@@ -0,0 +1,21 @@
+import { SessionPool } from 'crawlee';
+
+// Override the default Session pool configuration.
+const sessionPoolOptions = {
+ maxPoolSize: 100,
+};
+
+// Open Session Pool.
+const sessionPool = await SessionPool.open(sessionPoolOptions);
+
+// Get session.
+const session = await sessionPool.getSession();
+
+// Increase the errorScore.
+session.markBad();
+
+// Throw away the session.
+session.retire();
+
+// Lower the errorScore and mark the session good.
+session.markGood();
diff --git a/website/versioned_docs/version-3.7/guides/typescript_project.mdx b/website/versioned_docs/version-3.7/guides/typescript_project.mdx
new file mode 100644
index 000000000..db8c2452c
--- /dev/null
+++ b/website/versioned_docs/version-3.7/guides/typescript_project.mdx
@@ -0,0 +1,154 @@
+---
+id: typescript-project
+title: TypeScript Projects
+description: Stricter, safer, and better development experience
+---
+
+Crawlee is built with TypeScript, which means it provides the type definition directly in the package. This allows writing code with auto-completion for TypeScript and JavaScript code alike. Besides that, projects written in TypeScript can take advantage of compile-time type-checking and avoid many coding mistakes, while providing documentation for functions, parameters and return values. It will also help with refactoring a lot, and ensuring the least amount of bugs will sneak through.
+
+## Setting up a TypeScript project
+
+To use TypeScript in our projects, we'll need the following prerequisites:
+
+1. TypeScript compiler `tsc` installed somewhere:
+
+ ```shell script
+ npm install --save-dev typescript
+ ```
+
+ TypeScript can be a development dependency in our project, as shown above. There's no need to pollute the production environment or the system's global repository with TypeScript.
+
+2. A build script invoking `tsc` and a correctly specified `main` entry point defined in the `package.json` (pointing to the built code):
+
+ ```json
+ {
+ "scripts": {
+ "build": "tsc"
+ },
+ "main": "dist/main.js"
+ }
+ ```
+
+3. Type declarations for NodeJS, so we can take advantage of type-checking in all the features we'll use:
+
+ ```shell script
+ npm install --save-dev @types/node
+ ```
+
+4. TypeScript configuration file allowing `tsc` to understand the project layout and the features used in the project:
+
+ > We are extending the [`@apify/tsconfig`](https://github.com/apify/apify-tsconfig), it contains [the set of rules](https://github.com/apify/apify-tsconfig/blob/main/tsconfig.json) we believe are worth following.
+
+ > To be able to use feature called [Top level await](https://blog.saeloun.com/2021/11/25/ecmascript-top-level-await.html), we will need to set the `module` and `target` compiler options to `ES2022` or above. This will make the project compile to [ECMAScript Modules](https://nodejs.org/api/esm.html).
+
+ ```json title="tsconfig.json"
+ {
+ "extends": "@apify/tsconfig",
+ "compilerOptions": {
+ "module": "ES2022",
+ "target": "ES2022",
+ "outDir": "dist"
+ },
+ "include": [
+ "./src/**/*"
+ ]
+ }
+ ```
+
+ Place the content above inside a `tsconfig.json` in the root folder.
+
+ Also, to enjoy using the types in `.js` source files, VSCode users that are using JavaScript should create a `jsconfig.json` with the same content and add `"checkJs": true` to `"compilerOptions"`.
+
+ > If we want to use one of the browser crawlers, we will also need to add `"lib": ["DOM"]` to the compiler options.
+
+ Ensure that you have installed `@apify/tsconfig`
+
+ ```shell script
+ npm install --save-dev @apify/tsconfig
+ ```
+
+### Running the project with `ts-node`
+
+During development, it's handy to run the project directly instead of compiling the TypeScript code to JavaScript every time. We can use `ts-node` for that, just install it as a dev dependency and add a new NPM script:
+
+```shell script
+npm install --save-dev ts-node
+```
+
+> As mentioned above, our project will be compiled to use ES Modules. Because of this, we need to use the `ts-node-esm` binary.
+
+> We use the `-T` or `--transpileOnly` flag, this means the code will **not** be type-checked, which results in faster compilation. If you don't mind the added time and want to do the type checking, just remove this flag.
+
+```json title="package.json"
+{
+ "scripts": {
+ "start:dev": "ts-node-esm -T src/main.ts"
+ }
+}
+```
+
+### Running in production
+
+To run the project in production, we first need to compile it via build script. After that, we will have the compiled JavaScript code in the `dist`, and we can use `node dist/main.js` to run it.
+
+```json title="package.json"
+{
+ "scripts": {
+ "start:prod": "node dist/main.js"
+ }
+}
+```
+
+## Docker build
+
+For `Dockerfile` we recommend using multi-stage build, so we don't install the dev dependencies like TypeScript in the final image:
+
+```dockerfile title="Dockerfile"
+# using multistage build, as we need dev deps to build the TS source code
+FROM apify/actor-node:16 AS builder
+
+# copy all files, install all dependencies (including dev deps) and build the project
+COPY . ./
+RUN npm install --include=dev \
+ && npm run build
+
+# create final image
+FROM apify/actor-node:16
+# copy only necessary files
+COPY --from=builder /usr/src/app/package*.json ./
+COPY --from=builder /usr/src/app/dist ./dist
+
+# install only prod deps
+RUN npm --quiet set progress=false \
+ && npm install --only=prod --no-optional
+
+# run compiled code
+CMD npm run start:prod
+```
+
+### Putting it all together
+
+Let's wrap it up to. In addition to the scripts we described above, we also need to set the `type: 'module'` in the `package.json` to be able to use the Top level await described above. For convenience, we will have 3 `start` scripts, the default one will be an alias to `start:dev`, which is our `ts-node` script that does not require compilation (nor type checking). The production script (`start:prod`) is then used in the `Dockerfile`, after explicit `npm run build` call.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "type": "module",
+ "main": "dist/main.js",
+ "dependencies": {
+ "crawlee": "3.0.0"
+ },
+ "devDependencies": {
+ "@apify/tsconfig": "^0.1.0",
+ "@types/node": "^18.14.0",
+ "ts-node": "^10.8.0",
+ "typescript": "^4.7.4"
+ },
+ "scripts": {
+ "start": "npm run start:dev",
+ "start:prod": "node dist/main.js",
+ "start:dev": "ts-node-esm -T src/main.ts",
+ "build": "tsc"
+ }
+}
+```
diff --git a/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx
new file mode 100644
index 000000000..a69852298
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx
@@ -0,0 +1,92 @@
+---
+id: setting-up
+title: "Setting up"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ThemedImage from '@theme/ThemedImage';
+
+import ApiLink from '@site/src/components/ApiLink';
+
+To run Crawlee on your own computer, you need to meet the following pre-requisites first:
+
+1. Have **Node.js version 16.0** or higher installed.
+ - Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
+2. Have **NPM** installed, or use other package manager of your choice.
+ - NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
+
+If not certain, confirm the prerequisites by running:
+
+```bash
+node -v
+```
+
+```bash
+npm -v
+```
+
+## Creating a new project
+
+The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is embedded in the `crawlee` package:
+
+```bash
+npx crawlee create my-crawler
+```
+
+A prompt will be shown, asking you to choose a template. Crawlee is written in TypeScript so if you're familiar with it, choosing a TypeScript template will give you better code completion and static type checking, but feel free to use JavaScript as well. Functionally they're identical.
+
+Let's choose the first template called **Getting started example**. The command will create a new directory in your current working directory, called **my-crawler**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
+
+Let's try that!
+
+```bash
+cd my-crawler
+npm start
+```
+
+You will see log messages in the terminal as Crawlee boots up and starts scraping the Crawlee website.
+
+```log
+INFO PlaywrightCrawler: Starting the crawl
+INFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+```
+
+You can always terminate the crawl with a keypress in the terminal:
+
+```text
+CTRL+C
+```
+
+### Running headful browsers
+
+Browsers controlled by Playwright run headless (without a visible window). You can switch to headful by uncommenting the `headless: false` option in the crawler's constructor. This is useful in the development phase when you want to see what's going on in the browser.
+
+
+```typescript
+// Uncomment this option to see the browser window.
+headless: false
+```
+
+When you run the example again, after a second a Chromium browser window will open. In the window, you'll see quickly changing pages as the crawler does its job.
+
+:::note
+
+For the sake of this show off, we've slowed down the crawler, but rest assured, it's blazing fast in real world usage.
+
+:::
+
+<ThemedImage
+ alt="An image showing off Crawlee scraping the Crawlee website using Puppeteer/Playwright and Chromium"
+ sources={{
+ light: '/img/chrome-scrape-light.gif',
+ dark: '/img/chrome-scrape-dark.gif',
+ }}
+/>
+
+## Next lesson
+
+The next lesson will teach you how to create a very simple crawler and explain Crawlee components while building it.
diff --git a/website/versioned_docs/version-3.7/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.7/introduction/02-first-crawler.mdx
new file mode 100644
index 000000000..fd36ebedd
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/02-first-crawler.mdx
@@ -0,0 +1,121 @@
+---
+id: first-crawler
+title: "First crawler"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+In this lesson, you will build your first crawler. But before you do, let's briefly introduce the Crawlee classes involved in the process.
+
+## How Crawlee works
+
+There are 3 main crawler classes available for use in Crawlee. <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. We'll talk about their differences later. Now, let's talk about what they have in common.
+
+The general idea of each crawler is to go to a web page, open it, do some stuff there, save some results, continue to the next page, and repeat this process until the crawler's done its job. So the crawler always needs to find answers to two questions: **Where should I go?** and **What should I do there?** Answering those two questions is the only required setup. The crawlers have reasonable defaults for everything else.
+
+### The Where - `Request` and `RequestQueue`
+
+All crawlers use instances of the <ApiLink to="core/class/Request">`Request`</ApiLink> class to determine where they need to go. Each request may hold a lot of information, but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. Sometimes you have a pre-existing list of your own URLs that you wish to visit, perhaps a thousand. Other times you need to build this list dynamically as you crawl, adding more and more URLs to the list as you progress. Most of the time, you will use both options.
+
+The requests are stored in a <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, a dynamic queue of `Request` instances. You can seed it with start URLs and also add more requests while the crawler is running. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain, add them to the queue (called _enqueuing_) and repeat this process to build a queue of virtually unlimited number of URLs.
+
+### The What - `requestHandler`
+
+In the `requestHandler` you tell the crawler what to do at each and every page it visits. You can use it to handle extraction of data from the page, processing the data, saving it, calling APIs, doing calculations and so on.
+
+The `requestHandler` is a user-defined function, invoked automatically by the crawler for each `Request` from the `RequestQueue`. It always receives a single argument - a <ApiLink to="core/interface/CrawlingContext">`CrawlingContext`</ApiLink>. Its properties change depending on the crawler class used, but it always includes the `request` property, which represents the currently crawled URL and related metadata.
+
+## Building a crawler
+
+Let's put the theory into practice and start with something easy. Visit a page and get its HTML title. In this tutorial, you'll scrape the Crawlee website [https://crawlee.dev](https://crawlee.dev), but the same code will work for any website.
+
+:::info
+
+We are using a JavaScript feature called [Top level await](https://blog.saeloun.com/2021/11/25/ecmascript-top-level-await.html) in our examples. To be able to use that, you might need some extra setup. Namely, it requires the use of [ECMAScript Modules](https://nodejs.org/api/esm.html) - this means you either need to add `"type": "module"` to your `package.json` file, or use `*.mjs` extension for your files. Additionally, if you are in a TypeScript project, you need to set the `module` and `target` compiler options to `ES2022` or above.
+
+:::
+
+### Adding requests to the crawling queue
+
+Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
+
+```ts title="src/main.js"
+import { RequestQueue } from 'crawlee';
+
+// First you create the request queue instance.
+const requestQueue = await RequestQueue.open();
+// And then you add one or more requests to it.
+await requestQueue.addRequest({ url: 'https://crawlee.dev' });
+```
+
+The <ApiLink to="core/class/RequestQueue#addRequest">`requestQueue.addRequest()`</ApiLink> function automatically converts the object with URL string to a <ApiLink to="core/class/Request">`Request`</ApiLink> instance. So now you have a `requestQueue` that holds one request which points to `https://crawlee.dev`.
+
+:::tip
+
+The code above is for illustration of the request queue concept. Soon you'll learn about the `crawler.addRequests()` method which allows you to skip this initialization code, and it also supports adding a large number of requests without blocking.
+
+:::
+
+### Building a CheerioCrawler
+
+Crawlee comes with three main crawler classes: <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. You can read their short descriptions in the [Quick start](../quick-start) lesson.
+
+Unless you have a good reason to start with a different one, you should try building a `CheerioCrawler` first. It is an HTTP crawler with HTTP2 support, anti-blocking features and integrated HTML parser - [Cheerio](https://www.npmjs.com/package/cheerio). It's fast, simple, cheap to run and does not require complicated dependencies. The only downside is that it won't work out of the box for websites which require JavaScript rendering. But you might not need JavaScript rendering at all, because many modern websites use server-side rendering.
+
+Let's continue with the earlier `RequestQueue` example.
+
+```ts title="src/main.js"
+// Add import of CheerioCrawler
+import { RequestQueue, CheerioCrawler } from 'crawlee';
+
+const requestQueue = await RequestQueue.open();
+await requestQueue.addRequest({ url: 'https://crawlee.dev' });
+
+// Create the crawler and add the queue with our URL
+// and a request handler to process the page.
+const crawler = new CheerioCrawler({
+ requestQueue,
+ // The `$` argument is the Cheerio object
+ // which contains parsed HTML of the website.
+ async requestHandler({ $, request }) {
+ // Extract <title> text with Cheerio.
+ // See Cheerio documentation for API docs.
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ }
+})
+
+// Start the crawler and wait for it to finish
+await crawler.run();
+```
+
+When you run the example, you will see the title of https://crawlee.dev printed to the log. What really happens is that CheerioCrawler first makes an HTTP request to `https://crawlee.dev`, then parses the received HTML with Cheerio and makes it available as the `$` argument of the `requestHandler`.
+
+```log
+The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee.
+```
+
+### Add requests faster
+
+Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
+
+```ts title="src/main.js"
+// You don't need to import RequestQueue anymore
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ $, request }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ }
+})
+
+// Start the crawler with the provided URLs
+await crawler.run(['https://crawlee.dev']);
+```
+
+When you run this code, you'll see exactly the same output as with the earlier, longer example. The `RequestQueue` is still there, it's just managed by the crawler automatically.
+
+## Next lesson
+In the next lesson you'll learn about crawling links. That means finding new URLs on the pages you crawl and adding them to the `RequestQueue` for the crawler to visit.
diff --git a/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx
new file mode 100644
index 000000000..52f2a99f0
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx
@@ -0,0 +1,167 @@
+---
+id: adding-urls
+title: "Adding more URLs"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import FindWithELSource from '!!raw-loader!./03-find-el.ts';
+import FilterWithELSource from '!!raw-loader!./03-filter-el.ts';
+import FindWithoutELSource from '!!raw-loader!./03-find-without-el.ts';
+import FilterWithoutELSource from '!!raw-loader!./03-filter-without-el.ts';
+
+In the previous lesson you built a very simple crawler that downloads HTML of a single page, reads its title and prints
+it to the console. This is the original source code:
+
+```ts
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ $, request }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ }
+})
+
+await crawler.run(['https://crawlee.dev']);
+```
+
+In this lesson you'll use the example from the previous section and improve on it. You'll add more URLs to the queue and thanks to that the crawler will keep going, finding new links, enqueuing them into the `RequestQueue` and then scraping them.
+
+## How crawling works
+
+The process is simple:
+
+1. Find new links on the page.
+2. Filter only those pointing to the same domain, in this case `crawlee.dev`.
+3. Enqueue (add) them to the `RequestQueue`.
+4. Visit the newly enqueued links.
+5. Repeat the process.
+
+In the following paragraphs you will learn about the <ApiLink to="core/function/enqueueLinks">`enqueueLinks`</ApiLink> function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
+
+:::tip
+
+The `enqueueLinks` function is context aware. It means that it will read the information about the currently crawled page from the context, and you don't need to explicitly provide any arguments. It will find the links using the Cheerio function `$` and automatically add the links to the running crawler's `RequestQueue`.
+
+:::
+
+## Limit your crawls with `maxRequestsPerCrawl`
+
+ When you're just testing your code or when your crawler could potentially find millions of links, it's very useful to set a maximum limit of crawled pages. The option is called `maxRequestsPerCrawl`, is available in all crawlers, and you can set it like this:
+
+```ts
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ // ...
+});
+```
+
+This means that no new requests will be started after the 20th request is finished. The actual number of processed requests might be a little higher thanks to parallelization, because the running requests won't be forcefully aborted. It's not even possible in most cases.
+
+
+## Finding new links
+
+There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:
+
+```html
+<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>
+```
+
+Since this is the most common case, it is also the `enqueueLinks` default.
+
+<Tabs groupId="crawling-links">
+<TabItem value="enqueue-links" label="with enqueueLinks" default>
+<CodeBlock language="js" title="src/main.mjs">{FindWithELSource}</CodeBlock>
+</TabItem>
+<TabItem value="plain-js" label="without enqueueLinks">
+<CodeBlock language="js" title="src/main.mjs">{FindWithoutELSource}</CodeBlock>
+</TabItem>
+</Tabs>
+
+If you need to override the default selection of elements in `enqueueLinks`, you can use the `selector` argument.
+
+```ts
+await enqueueLinks({
+ selector: 'div.has-link'
+});
+```
+
+## Filtering links to same domain
+
+Websites typically contain a lot of links that lead away from the original page. This is normal, but when crawling a website, we usually want to crawl that one site and not let our crawler wander away to Google, Facebook and Twitter. Therefore, we need to filter out the off-domain links and only keep the ones that lead to the same domain.
+
+<Tabs groupId="crawling-links">
+ <TabItem value="enqueue-links" label="with enqueueLinks" default>
+ <CodeBlock language="js" title="src/main.mjs">{FilterWithELSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="plain-js" label="without enqueueLinks">
+ <CodeBlock language="js" title="src/main.mjs">{FilterWithoutELSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+The default behavior of `enqueueLinks` is to stay on the same hostname. This **does not include subdomains**. To include subdomains in your crawl, use the `strategy` argument.
+
+```ts
+await enqueueLinks({
+ strategy: 'same-domain'
+});
+```
+
+When you run the code, you will see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the first enqueued page and so on and so on.
+
+## Skipping duplicate URLs
+
+Skipping of duplicate URLs is critical, because visiting the same page multiple times would lead to duplicate results. This is automatically handled by the `RequestQueue` which deduplicates requests using their `uniqueKey`. This `uniqueKey` is automatically generated from the request's URL by lowercasing the URL, lexically ordering query parameters, removing fragments and a few other tweaks that ensure the queue only includes unique URLs.
+
+## Advanced filtering arguments
+
+While the defaults for `enqueueLinks` can be often exactly what you need, it also gives you fine-grained control over which URLs should be enqueued. One way we already mentioned above is by using the <ApiLink to="core/enum/EnqueueStrategy">`EnqueueStrategy`</ApiLink>. You can use the <ApiLink to="core/enum/EnqueueStrategy#All">`All`</ApiLink> strategy if you want to follow every single link, regardless of its domain, or you can enqueue links that target the same domain name with the <ApiLink to="core/enum/EnqueueStrategy#SameDomain">`SameDomain`</ApiLink> strategy.
+
+```ts
+await enqueueLinks({
+ strategy: 'all', // wander the internet
+});
+```
+
+### Filter URLs with patterns
+
+For even more control, you can use `globs`, `regexps` and `pseudoUrls` to filter the URLs. Each of those arguments is always an `Array`, but the contents can take on many forms. <ApiLink to="core/interface/EnqueueLinksOptions">See the reference</ApiLink> for more information about them as well as other options.
+
+:::caution
+
+If you provide one of those options, the default `same-hostname` strategy will **not** be applied unless explicitly set in the options.
+
+:::
+
+```ts
+await enqueueLinks({
+ globs: ['http?(s)://apify.com/*/*'],
+});
+```
+
+### Transform requests
+
+To have absolute control, we have the <ApiLink to="core/interface/EnqueueLinksOptions/#transformRequestFunction">`transformRequestFunction`</ApiLink>. Just before a new <ApiLink to="core/class/Request">`Request`</ApiLink> is constructed and enqueued to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, this function can be used to skip it or modify its contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+
+```ts
+await enqueueLinks({
+ globs: ['http?(s)://apify.com/*/*'],
+ transformRequestFunction(req) {
+ // ignore all links ending with `.pdf`
+ if (req.url.endsWith('.pdf')) return false;
+ return req;
+ },
+});
+```
+
+And that's it! `enqueueLinks()` is just one example of Crawlee's powerful helper functions. They're all designed to make your life easier, so you can focus on getting your data, while leaving the mundane crawling management to the tools.
+
+## Next lesson
+
+In the next lesson you will start your project of scraping a production website and learn some more Crawlee tricks in the process.
diff --git a/website/versioned_docs/version-3.7/introduction/03-filter-el.ts b/website/versioned_docs/version-3.7/introduction/03-filter-el.ts
new file mode 100644
index 000000000..9f9c09f73
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/03-filter-el.ts
@@ -0,0 +1,15 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ $, request, enqueueLinks }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ // The default behavior of enqueueLinks is to stay on the same hostname,
+ // so it does not require any parameters.
+ // This will ensure the subdomain stays the same.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.7/introduction/03-filter-without-el.ts
new file mode 100644
index 000000000..e50b05d8f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/03-filter-without-el.ts
@@ -0,0 +1,32 @@
+import { CheerioCrawler } from 'crawlee';
+import { URL } from 'node:url';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ request, $ }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+
+ const links = $('a[href]')
+ .map((_, el) => $(el).attr('href'))
+ .get();
+
+ // Besides resolving the URLs, we now also need to
+ // grab their hostname for filtering.
+ const { hostname } = new URL(request.loadedUrl);
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl),
+ );
+
+ // We use the hostname to filter links that point
+ // to a different domain, even subdomain.
+ const sameHostnameLinks = absoluteUrls
+ .filter((url) => url.hostname === hostname)
+ .map((url) => ({ url: url.href }));
+
+ // Finally, we have to add the URLs to the queue
+ await crawler.addRequests(sameHostnameLinks);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/introduction/03-find-el.ts b/website/versioned_docs/version-3.7/introduction/03-find-el.ts
new file mode 100644
index 000000000..50983a24f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/03-find-el.ts
@@ -0,0 +1,17 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Let's limit our crawls to make our
+ // tests shorter and safer.
+ maxRequestsPerCrawl: 20,
+ // enqueueLinks is an argument of the requestHandler
+ async requestHandler({ $, request, enqueueLinks }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ // The enqueueLinks function is context aware,
+ // so it does not require any parameters.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.7/introduction/03-find-without-el.ts
new file mode 100644
index 000000000..33561d019
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/03-find-without-el.ts
@@ -0,0 +1,29 @@
+import { CheerioCrawler } from 'crawlee';
+import { URL } from 'node:url';
+
+const crawler = new CheerioCrawler({
+ // Let's limit our crawls to make our
+ // tests shorter and safer.
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ request, $ }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+
+ // Without enqueueLinks, we first have to extract all
+ // the URLs from the page with Cheerio.
+ const links = $('a[href]')
+ .map((_, el) => $(el).attr('href'))
+ .get();
+
+ // Then we need to resolve relative URLs,
+ // otherwise they would be unusable for crawling.
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl).href,
+ );
+
+ // Finally, we have to add the URLs to the queue
+ await crawler.addRequests(absoluteUrls);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.7/introduction/04-pw-w-cheerio.ts
new file mode 100644
index 000000000..448d8fd2a
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/04-pw-w-cheerio.ts
@@ -0,0 +1,21 @@
+// Instead of CheerioCrawler let's use Playwright
+// to be able to render JavaScript.
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, parseWithCheerio }) => {
+ // Wait for the actor cards to render.
+ await page.waitForSelector('.collection-block-item');
+ // Extract the page's HTML from browser
+ // and parse it with Cheerio.
+ const $ = await parseWithCheerio();
+ // Use familiar Cheerio syntax to
+ // select all the actor cards.
+ $('.collection-block-item').each((i, el) => {
+ const text = $(el).text();
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
+ });
+ },
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.7/introduction/04-pw.ts b/website/versioned_docs/version-3.7/introduction/04-pw.ts
new file mode 100644
index 000000000..076094467
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/04-pw.ts
@@ -0,0 +1,21 @@
+// Instead of CheerioCrawler let's use Playwright
+// to be able to render JavaScript.
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page }) => {
+ // Wait for the actor cards to render.
+ await page.waitForSelector('.collection-block-item');
+ // Execute a function in the browser which targets
+ // the actor card elements and allows their manipulation.
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
+ // Extract text content from the actor cards
+ return els.map((el) => el.textContent);
+ });
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
+ });
+ },
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.7/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.7/introduction/04-real-world-project.mdx
new file mode 100644
index 000000000..d0c0c893b
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/04-real-world-project.mdx
@@ -0,0 +1,173 @@
+---
+id: real-world-project
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import PwSource from '!!raw-loader!./04-pw.ts';
+import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
+
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
+
+We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
+
+## Making a production-grade crawler
+
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
+
+The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
+
+:::tip
+
+If you're not interested in crawling theory, feel free to [skip to the next lesson](./crawling) and get right back to coding.
+
+:::
+
+## Drawing a plan
+
+Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
+
+## Choosing the data you need
+
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
+
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
+
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
+
+![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
+
+### The start URL(s)
+
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
+
+## Exploring the page
+
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
+
+### Categories and sorting
+
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
+
+:::caution
+
+Be careful, because on some websites, like [amazon.com](https://amazon.com), this is not true and the sum of products in categories is actually larger than what's available without filters. Learn more in our [tutorial on scraping websites with limited pagination](https://docs.apify.com/tutorials/scrape-paginated-sites).
+
+:::
+
+### Pagination
+
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
+
+```
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
+```
+
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+
+:::caution
+
+Similarly to the issue with filters explained above, the existence of pagination does not guarantee that you can simply paginate through all the results. Always test your assumptions about pagination. Otherwise, you might miss a chunk of results, and not even know about it.
+
+:::
+
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
+
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
+
+## The crawling strategy
+
+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
+
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
+
+`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
+
+## Sanity check
+
+Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
+
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
+
+<Tabs groupId="sanity-check">
+ <TabItem value="playwright" label="Playwright" default>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
+
+## DevTools - the scraper's toolbox
+
+:::info
+
+We'll use Chrome DevTools here, since it's the most common browser, but feel free to use any other, they're all very similar.
+
+:::
+
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
+
+## Selecting elements
+
+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.
+
+![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
+
+You'll see that you can select different elements inside the card. Instead, select the whole card, not just some of its contents, such as its title or description.
+
+![selected element](/img/getting-started/selected-element.png 'Selecting an element by hovering over it.')
+
+Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
+
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
+
+It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
+
+```ts
+document.querySelectorAll('.collection-block-item');
+```
+
+You will see that only the 31 collection cards will be returned, and nothing else.
+
+:::tip
+
+CSS selectors and DevTools are quite a big topic. If you want to learn more, visit the [Web scraping for beginners course](https://developers.apify.com/academy/web-scraping-for-beginners) in the Apify Academy. **It's free and open-source** ❤️.
+
+:::
+
+## Next lesson
+
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.7/introduction/05-crawling.mdx b/website/versioned_docs/version-3.7/introduction/05-crawling.mdx
new file mode 100644
index 000000000..a646699e5
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/05-crawling.mdx
@@ -0,0 +1,102 @@
+---
+id: crawling
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
+description: Your first steps into the world of scraping with Crawlee
+---
+
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
+
+## Crawling the listing pages
+
+In previous lessons, you used the `enqueueLinks()` function like this:
+
+```js
+await enqueueLinks();
+```
+
+While useful in that scenario, you need something different now. Instead of finding all the `<a href="..">` elements with links to the same hostname, you need to find only the specific ones that will take your crawler to the next page of results. Otherwise, the crawler will visit a lot of other pages that you're not interested in. Using the power of DevTools and yet another `enqueueLinks()` parameter, this becomes fairly easy.
+
+```ts
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
+ // otherwise enqueueLinks wouldn't enqueue anything.
+ await page.waitForSelector('.collection-block-item');
+
+ // Add links to the queue, but only from
+ // elements matching the provided selector.
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ },
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+```
+
+The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
+
+### The `selector` parameter of `enqueueLinks()`
+
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
+
+### The `label` of `enqueueLinks()`
+
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+
+## Crawling the detail pages
+
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+
+```ts
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ // We're not doing anything with the details yet.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+```
+
+The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
+
+## Next lesson
+
+This concludes the Crawling lesson, because you have taught the crawler to visit all the pages it needs. Let's continue with scraping data.
diff --git a/website/versioned_docs/version-3.7/introduction/06-example.ts b/website/versioned_docs/version-3.7/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.7/introduction/06-scraping.mdx b/website/versioned_docs/version-3.7/introduction/06-scraping.mdx
new file mode 100644
index 000000000..bf8b2d64f
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/06-scraping.mdx
@@ -0,0 +1,175 @@
+---
+id: scraping
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
+
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
+
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
+
+![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
+
+### Scraping the URL, Manufacturer and SKU
+
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
+
+:::info
+
+You can use `request.loadedUrl` as well. Remember the difference: `request.url` is what you enqueue, `request.loadedUrl` is what gets processed (after possible redirects).
+
+:::
+
+```js
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
+
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+```
+
+:::tip
+
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
+
+:::
+
+:::caution
+
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
+
+:::
+
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
+
+### Title
+
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
+
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
+
+:::tip
+
+Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of DevTools to open the search bar where you can quickly search for elements using their selectors. Always verify your scraping process and assumptions using the DevTools. It's faster than changing the crawler code all the time.
+
+:::
+
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
+
+```js
+const title = await page.locator('.product-meta h1').textContent();
+```
+
+### SKU
+
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
+
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
+
+```js
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+```
+
+### Current price
+
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
+
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
+
+```js
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+```
+
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
+
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
+
+### Stock available
+
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
+
+```js
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
+```
+
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
+
+And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
+
+```js
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
+```
+
+## Trying it out
+
+We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
+
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
+
+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
+
+```json
+{
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
+}
+```
+
+## Next lesson
+
+In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
diff --git a/website/versioned_docs/version-3.7/introduction/07-example.ts b/website/versioned_docs/version-3.7/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.7/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.7/introduction/07-saving-data.mdx
new file mode 100644
index 000000000..8f059d74c
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/07-saving-data.mdx
@@ -0,0 +1,59 @@
+---
+id: saving-data
+title: "Saving data"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
+
+A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
+
+First, add a new import to the top of the file:
+
+```ts
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+```
+
+Then, replace the `console.log(results)` call with:
+
+```js
+await Dataset.pushData(results);
+```
+
+and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
+
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
+
+## What's `Dataset.pushData()`
+
+&#8203;<ApiLink to="core/class/Dataset#pushData">`Dataset.pushData()`</ApiLink> is a function that saves data to the default <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>. `Dataset` is a storage designed to hold data in a format similar to a table. Each time you call `Dataset.pushData()` a new row in the table is created, with the property names serving as column titles. In the default configuration, the rows are represented as JSON files saved on your disk, but other storage systems can be plugged into Crawlee as well.
+
+:::info
+
+Each time you start Crawlee a default `Dataset` is automatically created, so there's no need to initialize it or create an instance first. You can create as many datasets as you want and even give them names. For more details see the [Result storage guide](../guides/result-storage#dataset) and the <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> function.
+
+:::
+
+## Finding saved data
+
+Unless you changed the configuration that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the `storage` directory that Crawlee creates in the working directory of the running script:
+
+```
+{PROJECT_FOLDER}/storage/datasets/default/
+```
+
+The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of `Dataset.pushData()` or one table row.
+
+:::tip
+
+If you would like to store your data in a single big file, instead of many small ones, see the [Result storage guide](../guides/result-storage#key-value-store) for Key-value stores.
+
+:::
+
+## Next lesson
+
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.7/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.7/introduction/08-refactoring.mdx
new file mode 100644
index 000000000..8c526cdcb
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/08-refactoring.mdx
@@ -0,0 +1,159 @@
+---
+id: refactoring
+title: 'Refactoring'
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Admonition from '@theme/Admonition';
+
+It may seem that the data is extracted and the crawler is done, but honestly, this is just the beginning. For the sake of brevity, we've completely omitted error handling, proxies, logging, architecture, tests, documentation and other stuff that a reliable software should have. The good thing is, **error handling is mostly done by Crawlee itself**, so no worries on that front, unless you need some custom magic.
+
+:::info
+
+If you've got to this point and are wondering where all the **anti-blocking, bot-protection avoiding stealthy features** are, you're right, we haven't shown you any. But that's the point! They are **automatically used** with the default configuration.
+
+That does not mean the default configuration can handle everything, but it should get you far enough. If you want to learn more, browse the [Avoid getting blocked](../guides/avoid-blocking), [Proxy management](../guides/proxy-management) and [Session management](../guides/session-management) guides.
+
+:::
+
+Anyway, to promote good coding practices, let's look at how you can use a <ApiLink to="core/class/Router">`Router`</ApiLink> to better structure your crawler code.
+
+## Routing
+
+In the following code we've made several changes:
+
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
+
+In our `main.mjs` file, we place the general structure of the crawler:
+
+```js title="src/main.mjs"
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+```
+
+Then in a separate `routes.mjs` file:
+
+```js title="src/routes.mjs"
+import { createPlaywrightRouter, Dataset } from 'crawlee';
+
+// createPlaywrightRouter() is only a helper to get better
+// intellisense and typings. You can use Router.create() too.
+export const router = createPlaywrightRouter();
+
+// This replaces the request.label === DETAIL branch of the if clause.
+router.addHandler('DETAIL', async ({ request, page, log }) => {
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
+ await Dataset.pushData(results);
+});
+
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
+// This is a fallback route which will handle the start URL
+// as well as the LIST labeled URLs.
+router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+});
+```
+
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.
+
+## Splitting your code into multiple files
+
+There's no reason not to split your code into multiple files and keep your logic separate. Less code in a single file means less code you need to think about at any time, and that's good. We would most likely go even further and split even the routes into separate files.
+
+## Using Crawlee `log` instead of `console.log`
+
+We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
+
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.
+
+## Using a router to structure your crawling
+
+At first, it might seem more readable using just a simple `if / else` statement to select different logic based on the crawled pages, but trust us, it becomes far less impressive when working with more than two different pages, and it definitely starts to fall apart when the logic to handle each page spans tens or hundreds of lines of code.
+
+It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
+
+## Next lesson
+
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.7/introduction/09-deployment.mdx b/website/versioned_docs/version-3.7/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.7/introduction/index.mdx b/website/versioned_docs/version-3.7/introduction/index.mdx
new file mode 100644
index 000000000..b64688861
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/index.mdx
@@ -0,0 +1,49 @@
+---
+id: introduction
+title: Introduction
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+Crawlee covers your crawling and scraping end-to-end and helps you **build reliable scrapers. Fast.**
+
+Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
+
+## What you will learn
+
+The goal of the introduction is to provide a step-by-step guide to the most important features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
+
+## 🛠 Features
+
+- Single interface for **HTTP and headless browser** crawling
+- Persistent **queue** for URLs to crawl (breadth & depth first)
+- Pluggable **storage** of both tabular data and files
+- Automatic **scaling** with available system resources
+- Integrated **proxy rotation** and session management
+- Lifecycles customizable with **hooks**
+- **CLI** to bootstrap your projects
+- Configurable **routing**, **error handling** and **retries**
+- **Dockerfiles** ready to deploy
+- Written in **TypeScript** with generics
+
+### 👾 HTTP crawling
+
+- Zero config **HTTP2 support**, even for proxies
+- Automatic generation of **browser-like headers**
+- Replication of browser **TLS fingerprints**
+- Integrated fast **HTML parsers**. Cheerio and JSDOM
+- Yes, you can scrape **JSON APIs** as well
+
+### 💻 Real browser crawling
+
+- JavaScript **rendering** and **screenshots**
+- **Headless** and **headful** support
+- Zero-config generation of **human-like fingerprints**
+- Automatic **browser management**
+- Use **Playwright** and **Puppeteer** with the same interface
+- **Chrome**, **Firefox**, **Webkit** and many others
+
+## Next lesson
+
+In the next lesson you will install Crawlee and learn how to bootstrap projects with the Crawlee CLI.
diff --git a/website/versioned_docs/version-3.7/introduction/tsconfig.json b/website/versioned_docs/version-3.7/introduction/tsconfig.json
new file mode 100644
index 000000000..62b847082
--- /dev/null
+++ b/website/versioned_docs/version-3.7/introduction/tsconfig.json
@@ -0,0 +1,21 @@
+{
+ "extends": "@apify/tsconfig",
+ "compilerOptions": {
+ "baseUrl": "../..",
+ "paths": {
+ "crawlee": ["packages/crawlee/src"],
+ "@crawlee/basic": ["packages/basic-crawler/src"],
+ "@crawlee/browser": ["packages/browser-crawler/src"],
+ "@crawlee/http": ["packages/http-crawler/src"],
+ "@crawlee/linkedom": ["packages/linkedom-crawler/src"],
+ "@crawlee/jsdom": ["packages/jsdom-crawler/src"],
+ "@crawlee/cheerio": ["packages/cheerio-crawler/src"],
+ "@crawlee/playwright": ["packages/playwright-crawler/src"],
+ "@crawlee/puppeteer": ["packages/puppeteer-crawler/src"],
+ "@crawlee/*": ["packages/*/src"]
+ },
+ "target": "ES2022",
+ "module": "ES2022",
+ "noUnusedParameters": false
+ }
+}
diff --git a/website/versioned_docs/version-3.7/quick-start/headful_playwright.ts b/website/versioned_docs/version-3.7/quick-start/headful_playwright.ts
new file mode 100644
index 000000000..088e5b7f2
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/headful_playwright.ts
@@ -0,0 +1,19 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
+ },
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/quick-start/headful_puppeteer.ts b/website/versioned_docs/version-3.7/quick-start/headful_puppeteer.ts
new file mode 100644
index 000000000..73798fd13
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/headful_puppeteer.ts
@@ -0,0 +1,19 @@
+import { PuppeteerCrawler, Dataset } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
+ },
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/quick-start/index.mdx b/website/versioned_docs/version-3.7/quick-start/index.mdx
new file mode 100644
index 000000000..ffe401dcd
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/index.mdx
@@ -0,0 +1,171 @@
+---
+id: quick-start
+title: Quick Start
+description: With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+
+import Admonition from '@theme/Admonition';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+import ThemedImage from '@theme/ThemedImage';
+
+import CheerioSource from '!!raw-loader!roa-loader!./quick_start_cheerio.ts';
+import PlaywrightSource from '!!raw-loader!roa-loader!./quick_start_playwright.ts';
+import PlaywrightHeadful from '!!raw-loader!roa-loader!./headful_playwright.ts';
+import PuppeteerSource from '!!raw-loader!roa-loader!./quick_start_puppeteer.ts';
+import PuppeteerHeadful from '!!raw-loader!roa-loader!./headful_puppeteer.ts';
+
+import CheerioLog from '!!raw-loader!./quick_start_cheerio.txt';
+
+With this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](./introduction), which is a comprehensive step-by-step guide for creating your first scraper.
+
+## Choose your crawler
+
+Crawlee comes with three main crawler classes: <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. All classes share the same interface for maximum flexibility when switching between them.
+
+### CheerioCrawler
+This is a plain HTTP crawler. It parses HTML using the [Cheerio](https://github.com/cheeriojs/cheerio) library and crawls the web using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client which masks as a browser. It's very fast and efficient, but can't handle JavaScript rendering.
+
+### PuppeteerCrawler
+This crawler uses a headless browser to crawl, controlled by the [Puppeteer](https://github.com/puppeteer/puppeteer) library. It can control Chromium or Chrome. Puppeteer is the de-facto standard in headless browser automation.
+
+### PlaywrightCrawler
+[Playwright](https://github.com/microsoft/playwright) is a more powerful and full-featured successor to Puppeteer. It can control Chromium, Chrome, Firefox, Webkit and many other browsers. If you're not familiar with Puppeteer already, and you need a headless browser, go with Playwright.
+
+:::caution before you start
+
+Crawlee requires [Node.js 16 or later](https://nodejs.org/en/).
+
+:::
+
+## Installation with Crawlee CLI
+
+The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**.
+The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
+
+```bash
+npx crawlee create my-crawler
+```
+
+After the installation is complete you can start the crawler like this:
+
+```bash
+cd my-crawler && npm start
+```
+
+## Manual installation
+
+You can add Crawlee to any Node.js project by running:
+
+<Tabs groupId="quick_start">
+<TabItem value="cheerio" label="CheerioCrawler" default>
+<CodeBlock language="bash">npm install crawlee</CodeBlock>
+</TabItem>
+<TabItem value="playwright" label="PlaywrightCrawler">
+
+:::caution
+
+`playwright` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM. 👇
+
+:::
+
+<CodeBlock language="bash">npm install crawlee playwright</CodeBlock>
+</TabItem>
+<TabItem value="puppeteer" label="PuppeteerCrawler">
+
+:::caution
+
+`puppeteer` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM. 👇
+
+:::
+
+<CodeBlock language="bash">npm install crawlee puppeteer</CodeBlock>
+</TabItem>
+</Tabs>
+
+## Crawling
+
+Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler.
+
+<Admonition type="caution" title="Don't forget about module imports">
+ To run the example, add a <code>"type": "module"</code> clause into your <code>package.json</code> or
+ copy it into a file with an <code>.mjs</code> suffix. This enables <code>import</code> statements in Node.js.
+ See <a href="https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#enabling" target="_blank" rel="noreferrer">Node.js docs</a> for
+ more information.
+</Admonition>
+
+<Tabs groupId="quick_start">
+ <TabItem value="cheerio" label="CheerioCrawler" default>
+ <RunnableCodeBlock className="language-js" type="cheerio">{CheerioSource}</RunnableCodeBlock>
+ </TabItem>
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <RunnableCodeBlock className="language-js" type="playwright">{PlaywrightSource}</RunnableCodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <RunnableCodeBlock className="language-js" type="puppeteer">{PuppeteerSource}</RunnableCodeBlock>
+ </TabItem>
+</Tabs>
+
+When you run the example, you will see Crawlee automating the data extraction process in your terminal.
+
+<CodeBlock language="log">{CheerioLog}</CodeBlock>
+
+
+### Running headful browsers
+
+Browsers controlled by Puppeteer and Playwright run headless (without a visible window). You can switch to headful by adding the `headless: false` option to the crawlers' constructor. This is useful in the development phase when you want to see what's going on in the browser.
+
+<Tabs groupId="quick_start">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <RunnableCodeBlock className="language-js" type="playwright">{PlaywrightHeadful}</RunnableCodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <RunnableCodeBlock className="language-js" type="puppeteer">{PuppeteerHeadful}</RunnableCodeBlock>
+ </TabItem>
+</Tabs>
+
+When you run the example code, you'll see an automated browser blaze through the Crawlee website.
+
+:::note
+
+For the sake of this show off, we've slowed down the crawler, but rest assured, it's blazing fast in real world usage.
+
+:::
+
+<ThemedImage
+ alt="An image showing off Crawlee scraping the Crawlee website using Puppeteer/Playwright and Chromium"
+ sources={{
+ light: '/img/chrome-scrape-light.gif',
+ dark: '/img/chrome-scrape-dark.gif',
+ }}
+/>
+
+## Results
+
+Crawlee stores data to the `./storage` directory in your current working directory. The results of your crawl will be available under `./storage/datasets/default/*.json` as JSON files.
+
+```json title="./storage/datasets/default/000000001.json"
+{
+ "url": "https://crawlee.dev/",
+ "title": "Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee"
+}
+```
+
+:::tip
+
+You can override the storage directory by setting the `CRAWLEE_STORAGE_DIR` environment variable.
+
+:::
+
+## Examples and further reading
+You can find more examples showcasing various features of Crawlee in the [Examples](./examples) section of the documentation. To better understand Crawlee and its components you should read the [Introduction](./introduction) step-by-step guide.
+
+**Related links**
+
+- [Configuration](./guides/configuration)
+- [Request storage](./guides/request-storage)
+- [Result storage](./guides/result-storage)
diff --git a/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.ts b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.ts
new file mode 100644
index 000000000..62fe06f9d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.ts
@@ -0,0 +1,24 @@
+import { CheerioCrawler, Dataset } from 'crawlee';
+
+// CheerioCrawler crawls the web using HTTP requests
+// and parses HTML using the Cheerio library.
+const crawler = new CheerioCrawler({
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, $, enqueueLinks, log }) {
+ const title = $('title').text();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
+
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt
new file mode 100644
index 000000000..e68fa4133
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt
@@ -0,0 +1,5 @@
+INFO CheerioCrawler: Starting the crawl
+INFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'
diff --git a/website/versioned_docs/version-3.7/quick-start/quick_start_playwright.ts b/website/versioned_docs/version-3.7/quick-start/quick_start_playwright.ts
new file mode 100644
index 000000000..00bf910dc
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/quick_start_playwright.ts
@@ -0,0 +1,26 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
+const crawler = new PlaywrightCrawler({
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
+
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
+ },
+ // Uncomment this option to see the browser window.
+ // headless: false,
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/quick-start/quick_start_puppeteer.ts b/website/versioned_docs/version-3.7/quick-start/quick_start_puppeteer.ts
new file mode 100644
index 000000000..5b8bb70e6
--- /dev/null
+++ b/website/versioned_docs/version-3.7/quick-start/quick_start_puppeteer.ts
@@ -0,0 +1,26 @@
+import { PuppeteerCrawler, Dataset } from 'crawlee';
+
+// PuppeteerCrawler crawls the web using a headless
+// browser controlled by the Puppeteer library.
+const crawler = new PuppeteerCrawler({
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
+
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
+ },
+ // Uncomment this option to see the browser window.
+ // headless: false,
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.7/upgrading/upgrading_v1.md b/website/versioned_docs/version-3.7/upgrading/upgrading_v1.md
new file mode 100644
index 000000000..c6d7d6bba
--- /dev/null
+++ b/website/versioned_docs/version-3.7/upgrading/upgrading_v1.md
@@ -0,0 +1,450 @@
+---
+id: upgrading-to-v1
+title: Upgrading to v1
+---
+
+## Summary
+After 3.5 years of rapid development and a lot of breaking changes and deprecations,
+here comes the result - **Apify SDK v1**. There were two goals for this release. **Stability**
+and **adding support for more browsers** - Firefox and Webkit (Safari).
+
+The SDK has grown quite popular over the years, powering thousands of web scraping
+and automation projects. We think our developers deserve a stable environment to work
+in and by releasing SDK v1, **we commit to only make breaking changes once a year,
+with a new major release**.
+
+We added support for more browsers by replacing `PuppeteerPool` with
+[`browser-pool`](https://github.com/apify/browser-pool). A new library that we created
+specifically for this purpose. It builds on the ideas from `PuppeteerPool` and extends
+them to support [Playwright](https://github.com/microsoft/playwright). Playwright is
+a browser automation library similar to Puppeteer. It works with all well known browsers
+and uses almost the same interface as Puppeteer, while adding useful features and simplifying
+common tasks. Don't worry, you can still use Puppeteer with the new `BrowserPool`.
+
+A large breaking change is that neither `puppeteer` nor `playwright` are bundled with
+the SDK v1. To make the choice of a library easier and installs faster, users will
+have to install the selected modules and versions themselves. This allows us to add
+support for even more libraries in the future.
+
+Thanks to the addition of Playwright we now have a `PlaywrightCrawler`. It is very similar
+to `PuppeteerCrawler` and you can pick the one you prefer. It also means we needed to make
+some interface changes. The `launchPuppeteerFunction` option of `PuppeteerCrawler` is gone
+and `launchPuppeteerOptions` were replaced by `launchContext`. We also moved things around
+in the `handlePageFunction` arguments. See the
+[migration guide](#migration-guide)
+for more detailed explanation and migration examples.
+
+What's in store for SDK v2? We want to split the SDK into smaller libraries,
+so that everyone can install only the things they need. We plan a TypeScript migration
+to make crawler development faster and safer. Finally, we will take a good look
+at the interface of the whole SDK and update it to improve the developer experience.
+Bug fixes and scraping features will of course keep landing in versions 1.X as well.
+
+## Migration Guide
+There are a lot of breaking changes in the v1.0.0 release, but we're confident that
+updating your code will be a matter of minutes. Below, you'll find examples how to do it
+and also short tutorials how to use many of the new features.
+
+> Many of the new features are made with power users in mind,
+> so don't worry if something looks complicated. You don't need to use it.
+
+## Installation
+Previous versions of the SDK bundled the `puppeteer` package, so you did not have to install
+it. SDK v1 supports also `playwright` and we don't want to force users to install both.
+To install SDK v1 with Puppeteer (same as previous versions), run:
+
+```bash
+npm install apify puppeteer
+```
+
+To install SDK v1 with Playwright run:
+```bash
+npm install apify playwright
+```
+
+> While we tried to add the most important functionality in the initial release,
+> you may find that there are still some utilities or options that are only
+> supported by Puppeteer and not Playwright.
+
+## Running on Apify Platform
+If you want to make use of Playwright on the Apify Platform, you need to use a Docker image
+that supports Playwright. We've created them for you, so head over to the new
+[Docker image guide](https://sdk.apify.com/docs/guides/docker-images) and pick the one
+that best suits your needs.
+
+Note that your `package.json` **MUST** include `puppeteer` and/or `playwright` as dependencies.
+If you don't list them, the libraries will be uninstalled from your `node_modules` folder
+when you build your actors.
+
+## Handler arguments are now Crawling Context
+Previously, arguments of user provided handler functions were provided in separate
+objects. This made it difficult to track values across function invocations.
+
+```js
+const handlePageFunction = async (args1) => {
+ args1.hasOwnProperty('proxyInfo') // true
+}
+
+const handleFailedRequestFunction = async (args2) => {
+ args2.hasOwnProperty('proxyInfo') // false
+}
+
+args1 === args2 // false
+```
+
+This happened because a new arguments object was created for each function.
+With SDK v1 we now have a single object called Crawling Context.
+
+```js
+const handlePageFunction = async (crawlingContext1) => {
+ crawlingContext1.hasOwnProperty('proxyInfo') // true
+}
+
+const handleFailedRequestFunction = async (crawlingContext2) => {
+ crawlingContext2.hasOwnProperty('proxyInfo') // true
+}
+
+// All contexts are the same object.
+crawlingContext1 === crawlingContext2 // true
+```
+
+### `Map` of crawling contexts and their IDs
+Now that all the objects are the same, we can keep track of all running crawling contexts.
+We can do that by working with the new `id` property of `crawlingContext`
+This is useful when you need cross-context access.
+
+```js
+let masterContextId;
+const handlePageFunction = async ({ id, page, request, crawler }) => {
+ if (request.userData.masterPage) {
+ masterContextId = id;
+ // Prepare the master page.
+ } else {
+ const masterContext = crawler.crawlingContexts.get(masterContextId);
+ const masterPage = masterContext.page;
+ const masterRequest = masterContext.request;
+ // Now we can manipulate the master data from another handlePageFunction.
+ }
+}
+```
+
+### `autoscaledPool` was moved under `crawlingContext.crawler`
+To prevent bloat and to make access to certain key objects easier, we exposed a `crawler`
+property on the handle page arguments.
+
+```js
+const handePageFunction = async ({ request, page, crawler }) => {
+ await crawler.requestQueue.addRequest({ url: 'https://example.com' });
+ await crawler.autoscaledPool.pause();
+}
+```
+
+This also means that some shorthands like `puppeteerPool` or `autoscaledPool` were
+no longer necessary.
+
+```js
+const handePageFunction = async (crawlingContext) => {
+ crawlingContext.autoscaledPool // does NOT exist anymore
+ crawlingContext.crawler.autoscaledPool // <= this is correct usage
+}
+```
+
+## Replacement of `PuppeteerPool` with `BrowserPool`
+`BrowserPool` was created to extend `PuppeteerPool` with the ability to manage other
+browser automation libraries. The API is similar, but not the same.
+
+### Access to running `BrowserPool`
+Only `PuppeteerCrawler` and `PlaywrightCrawler` use `BrowserPool`. You can access it
+on the `crawler` object.
+
+```js
+const crawler = new Apify.PlaywrightCrawler({
+ handlePageFunction: async ({ page, crawler }) => {
+ crawler.browserPool // <-----
+ }
+});
+
+crawler.browserPool // <-----
+```
+
+### Pages now have IDs
+And they're equal to `crawlingContext.id` which gives you access to full `crawlingContext`
+in hooks. See [Lifecycle hooks](#configuration-and-lifecycle-hooks) below.
+
+```js
+const pageId = browserPool.getPageId
+```
+
+### Configuration and lifecycle hooks
+The most important addition with `BrowserPool` are the
+[lifecycle hooks](https://github.com/apify/browser-pool#browserpool).
+You can access them via `browserPoolOptions` in both crawlers. A full list of `browserPoolOptions`
+can be found in [`browser-pool` readme](https://github.com/apify/browser-pool#new-browserpooloptions).
+
+```js
+const crawler = new Apify.PuppeteerCrawler({
+ browserPoolOptions: {
+ retireBrowserAfterPageCount: 10,
+ preLaunchHooks: [
+ async (pageId, launchContext) => {
+ const { request } = crawler.crawlingContexts.get(pageId);
+ if (request.userData.useHeadful === true) {
+ launchContext.launchOptions.headless = false;
+ }
+ }
+ ]
+ }
+})
+```
+
+### Introduction of `BrowserController`
+[`BrowserController`](https://github.com/apify/browser-pool#browsercontroller)
+is a class of `browser-pool` that's responsible for browser management.
+Its purpose is to provide a single API for working with both Puppeteer and Playwright browsers.
+It works automatically in the background, but if you ever wanted to close a browser properly,
+you should use a `browserController` to do it. You can find it in the handle page arguments.
+
+```js
+const handlePageFunction = async ({ page, browserController }) => {
+ // Wrong usage. Could backfire because it bypasses BrowserPool.
+ await page.browser().close();
+
+ // Correct usage. Allows graceful shutdown.
+ await browserController.close();
+
+ const cookies = [/* some cookie objects */];
+ // Wrong usage. Will only work in Puppeteer and not Playwright.
+ await page.setCookies(...cookies);
+
+ // Correct usage. Will work in both.
+ await browserController.setCookies(page, cookies);
+}
+```
+
+The `BrowserController` also includes important information about the browser, such as
+the context it was launched with. This was difficult to do before SDK v1.
+
+```js
+const handlePageFunction = async ({ browserController }) => {
+ // Information about the proxy used by the browser
+ browserController.launchContext.proxyInfo
+
+ // Session used by the browser
+ browserController.launchContext.session
+}
+```
+
+### `BrowserPool` methods vs `PuppeteerPool`
+Some functions were removed (in line with earlier deprecations), and some were changed a bit:
+
+```js
+// OLD
+await puppeteerPool.recyclePage(page);
+
+// NEW
+await page.close();
+```
+
+```js
+// OLD
+await puppeteerPool.retire(page.browser());
+
+// NEW
+browserPool.retireBrowserByPage(page);
+```
+
+```js
+// OLD
+await puppeteerPool.serveLiveViewSnapshot();
+
+// NEW
+// There's no LiveView in BrowserPool
+```
+
+## Updated `PuppeteerCrawlerOptions`
+To keep `PuppeteerCrawler` and `PlaywrightCrawler` consistent, we updated the options.
+
+### Removal of `gotoFunction`
+The concept of a configurable `gotoFunction` is not ideal. Especially since we use a modified
+`gotoExtended`. Users have to know this when they override `gotoFunction` if they want to
+extend default behavior. We decided to replace `gotoFunction` with `preNavigationHooks` and
+`postNavigationHooks`.
+
+The following example illustrates how `gotoFunction` makes things complicated.
+```js
+const gotoFunction = async ({ request, page }) => {
+ // pre-processing
+ await makePageStealthy(page);
+
+ // Have to remember how to do this:
+ const response = await gotoExtended(page, request, {/* have to remember the defaults */});
+
+ // post-processing
+ await page.evaluate(() => {
+ window.foo = 'bar';
+ });
+
+ // Must not forget!
+ return response;
+}
+
+const crawler = new Apify.PuppeteerCrawler({
+ gotoFunction,
+ // ...
+})
+```
+
+With `preNavigationHooks` and `postNavigationHooks` it's much easier. `preNavigationHooks`
+are called with two arguments: `crawlingContext` and `gotoOptions`. `postNavigationHooks`
+are called only with `crawlingContext`.
+
+```js
+const preNavigationHooks = [
+ async ({ page }) => makePageStealthy(page)
+];
+
+const postNavigationHooks = [
+ async ({ page }) => page.evaluate(() => {
+ window.foo = 'bar'
+ })
+]
+
+const crawler = new Apify.PuppeteerCrawler({
+ preNavigationHooks,
+ postNavigationHooks,
+ // ...
+})
+```
+
+### `launchPuppeteerOptions` => `launchContext`
+Those were always a point of confusion because they merged custom Apify options with
+`launchOptions` of Puppeteer.
+
+```js
+const launchPuppeteerOptions = {
+ useChrome: true, // Apify option
+ headless: false, // Puppeteer option
+}
+```
+
+Use the new `launchContext` object, which explicitly defines `launchOptions`.
+`launchPuppeteerOptions` were removed.
+
+```js
+const crawler = new Apify.PuppeteerCrawler({
+ launchContext: {
+ useChrome: true, // Apify option
+ launchOptions: {
+ headless: false // Puppeteer option
+ }
+ }
+})
+```
+
+> LaunchContext is also a type of [`browser-pool`](https://github.com/apify/browser-pool) and
+> the structure is exactly the same there. SDK only adds extra options.
+
+### Removal of `launchPuppeteerFunction`
+`browser-pool` introduces the idea of [lifecycle hooks](https://github.com/apify/browser-pool#browserpool),
+which are functions that are executed when a certain event in the browser lifecycle happens.
+
+```js
+const launchPuppeteerFunction = async (launchPuppeteerOptions) => {
+ if (someVariable === 'chrome') {
+ launchPuppeteerOptions.useChrome = true;
+ }
+ return Apify.launchPuppeteer(launchPuppeteerOptions);
+}
+
+const crawler = new Apify.PuppeteerCrawler({
+ launchPuppeteerFunction,
+ // ...
+})
+```
+
+Now you can recreate the same functionality with a `preLaunchHook`:
+
+```js
+const maybeLaunchChrome = (pageId, launchContext) => {
+ if (someVariable === 'chrome') {
+ launchContext.useChrome = true;
+ }
+}
+
+const crawler = new Apify.PuppeteerCrawler({
+ browserPoolOptions: {
+ preLaunchHooks: [maybeLaunchChrome]
+ },
+ // ...
+})
+```
+
+This is better in multiple ways. It is consistent across both Puppeteer and Playwright.
+It allows you to easily construct your browsers with pre-defined behavior:
+
+```js
+const preLaunchHooks = [
+ maybeLaunchChrome,
+ useHeadfulIfNeeded,
+ injectNewFingerprint,
+]
+```
+
+And thanks to the addition of [`crawler.crawlingContexts`](#handler-arguments-are-now-crawling-context)
+the functions also have access to the `crawlingContext` of the `request` that triggered the launch.
+
+```js
+const preLaunchHooks = [
+ async function maybeLaunchChrome(pageId, launchContext) {
+ const { request } = crawler.crawlingContexts.get(pageId);
+ if (request.userData.useHeadful === true) {
+ launchContext.launchOptions.headless = false;
+ }
+ }
+]
+```
+
+## Launch functions
+In addition to `Apify.launchPuppeteer()` we now also have `Apify.launchPlaywright()`.
+
+### Updated arguments
+We [updated the launch options object](#launchpuppeteeroptions--launchcontext) because
+it was a frequent source of confusion.
+
+```js
+// OLD
+await Apify.launchPuppeteer({
+ useChrome: true,
+ headless: true,
+})
+
+// NEW
+await Apify.launchPuppeteer({
+ useChrome: true,
+ launchOptions: {
+ headless: true,
+ }
+})
+```
+
+### Custom modules
+`Apify.launchPuppeteer` already supported the `puppeteerModule` option. With Playwright,
+we normalized the name to `launcher` because the `playwright` module itself does not
+launch browsers.
+
+```js
+const puppeteer = require('puppeteer');
+const playwright = require('playwright');
+
+await Apify.launchPuppeteer();
+// Is the same as:
+await Apify.launchPuppeteer({
+ launcher: puppeteer
+})
+
+await Apify.launchPlaywright();
+// Is the same as:
+await Apify.launchPlaywright({
+ launcher: playwright.chromium
+})
+```
diff --git a/website/versioned_docs/version-3.7/upgrading/upgrading_v2.md b/website/versioned_docs/version-3.7/upgrading/upgrading_v2.md
new file mode 100644
index 000000000..c0a84584d
--- /dev/null
+++ b/website/versioned_docs/version-3.7/upgrading/upgrading_v2.md
@@ -0,0 +1,8 @@
+---
+id: upgrading-to-v2
+title: Upgrading to v2
+---
+
+- **BREAKING**: Require Node.js >=15.10.0 because HTTP2 support on lower Node.js versions is very buggy.
+- **BREAKING**: Bump `cheerio` to `1.0.0-rc.10` from `rc.3`. There were breaking changes in `cheerio` between the versions so this bump might be breaking for you as well.
+- Remove `LiveViewServer` which was deprecated before release of SDK v1.
diff --git a/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md
new file mode 100644
index 000000000..bd7f43018
--- /dev/null
+++ b/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md
@@ -0,0 +1,484 @@
+---
+id: upgrading-to-v3
+title: Upgrading to v3
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+This page summarizes most of the breaking changes between Crawlee (v3) and Apify SDK (v2). Crawlee is the spiritual successor to Apify SDK, so we decided to keep the versioning and release Crawlee as v3.
+
+:::info Crawlee vs Apify SDK v2
+
+Up until version 3 of `apify`, the package contained both scraping related tools and Apify platform related helper methods. With v3 we are splitting the whole project into two main parts:
+
+- [Crawlee](https://github.com/apify/crawlee), the new web-scraping library, available as [`crawlee`](https://www.npmjs.com/package/crawlee) package on NPM
+- [Apify SDK](https://github.com/apify/apify-sdk-js), helpers for the Apify platform, available as [`apify`](https://www.npmjs.com/package/apify) package on NPM
+
+:::
+
+## Crawlee monorepo
+
+The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:
+
+- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes
+- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`
+- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`
+- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`
+- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`
+- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`
+- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))
+- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))
+- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative
+- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package
+- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods
+- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)
+
+### Installing Crawlee
+
+Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. `@crawlee/playwright` if you plan on using `playwright` - it already contains everything from the `@crawlee/browser` package, which includes everything from `@crawlee/basic`, which includes everything from `@crawlee/core`.
+
+If we don't care much about additional code being pulled in, we can just use the `crawlee` meta-package, which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
+
+```bash
+npm install crawlee
+```
+
+Or if all we need is cheerio support, we can install only `@crawlee/cheerio`.
+
+```bash
+npm install @crawlee/cheerio
+```
+
+When using `playwright` or `puppeteer`, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.
+
+```bash
+npm install crawlee playwright
+# or npm install @crawlee/playwright playwright
+```
+
+Alternatively we can also use the `crawlee` meta-package which contains (re-exports) most of the `@crawlee/*` packages, and therefore contains all the crawler classes.
+
+> Sometimes you might want to use some utility methods from `@crawlee/utils`, so you might want to install that as well. This package contains some utilities that were previously available under `Apify.utils`. Browser related utilities can be also found in the crawler packages (e.g. `@crawlee/playwright`).
+
+## Full TypeScript support
+
+Both Crawlee and Apify SDK are full TypeScript rewrite, so they include up-to-date types in the package. For your TypeScript crawlers we recommend using our predefined TypeScript configuration from `@apify/tsconfig` package. Don't forget to set the `module` and `target` to `ES2022` or above to be able to use top level await.
+
+> The `@apify/tsconfig` config has [`noImplicitAny`](https://www.typescriptlang.org/tsconfig#noImplicitAny) enabled, you might want to disable it during the initial development as it will cause build failures if you left some unused local variables in your code.
+
+```json title="tsconfig.json"
+{
+ "extends": "@apify/tsconfig",
+ "compilerOptions": {
+ "module": "ES2022",
+ "target": "ES2022",
+ "outDir": "dist",
+ "lib": ["DOM"]
+ },
+ "include": [
+ "./src/**/*"
+ ]
+}
+```
+
+### Docker build
+
+For `Dockerfile` we recommend using multi-stage build, so you don't install the dev dependencies like TypeScript in your final image:
+
+```dockerfile title="Dockerfile"
+# using multistage build, as we need dev deps to build the TS source code
+FROM apify/actor-node:16 AS builder
+
+# copy all files, install all dependencies (including dev deps) and build the project
+COPY . ./
+RUN npm install --include=dev \
+ && npm run build
+
+# create final image
+FROM apify/actor-node:16
+# copy only necessary files
+COPY --from=builder /usr/src/app/package*.json ./
+COPY --from=builder /usr/src/app/README.md ./
+COPY --from=builder /usr/src/app/dist ./dist
+COPY --from=builder /usr/src/app/apify.json ./apify.json
+COPY --from=builder /usr/src/app/INPUT_SCHEMA.json ./INPUT_SCHEMA.json
+
+# install only prod deps
+RUN npm --quiet set progress=false \
+ && npm install --only=prod --no-optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --only=prod --no-optional --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# run compiled code
+CMD npm run start:prod
+```
+
+## Browser fingerprints
+
+Previously we had a magical `stealth` option in the puppeteer crawler that enabled several tricks aiming to mimic the real users as much as possible. While this worked to a certain degree, we decided to replace it with generated browser fingerprints.
+
+In case we don't want to have dynamic fingerprints, we can disable this behaviour via `useFingerprints` in `browserPoolOptions`:
+
+ ```ts
+const crawler = new PlaywrightCrawler({
+ browserPoolOptions: {
+ useFingerprints: false,
+ },
+});
+ ```
+
+## Session cookie method renames
+
+Previously, if we wanted to get or add cookies for the session that would be used for the request, we had to call `session.getPuppeteerCookies()` or `session.setPuppeteerCookies()`. Since this method could be used for any of our crawlers, not just `PuppeteerCrawler`, the methods have been renamed to `session.getCookies()` and `session.setCookies()` respectively. Otherwise, their usage is exactly the same!
+
+## Memory storage
+
+When we store some data or intermediate state (like the one `RequestQueue` holds), we now use `@crawlee/memory-storage` by default. It is an alternative to the `@apify/storage-local`, that stores the state inside memory (as opposed to SQLite database used by `@apify/storage-local`). While the state is stored in memory, it also dumps it to the file system, so we can observe it, as well as respects the existing data stored in KeyValueStore (e.g. the `INPUT.json` file).
+
+When we want to run the crawler on Apify platform, we need to use `Actor.init` or `Actor.main`, which will automatically switch the storage client to `ApifyClient` when on the Apify platform.
+
+We can still use the `@apify/storage-local`, to do it, first install it pass it to the `Actor.init` or `Actor.main` options:
+
+> `@apify/storage-local` v2.1.0+ is required for Crawlee
+
+```ts
+import { Actor } from 'apify';
+import { ApifyStorageLocal } from '@apify/storage-local';
+
+const storage = new ApifyStorageLocal(/* options like `enableWalMode` belong here */);
+await Actor.init({ storage });
+```
+
+## Purging of the default storage
+
+Previously the state was preserved between local runs, and we had to use `--purge` argument of the `apify-cli`. With Crawlee, this is now the default behaviour, we purge the storage automatically on `Actor.init/main` call. We can opt out of it via `purge: false` in the `Actor.init` options.
+
+## Renamed crawler options and interfaces
+
+Some options were renamed to better reflect what they do. We still support all the old parameter names too, but not at the TS level.
+
+* `handleRequestFunction` -> `requestHandler`
+* `handlePageFunction` -> `requestHandler`
+* `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs`
+* `handlePageTimeoutSecs` -> `requestHandlerTimeoutSecs`
+* `requestTimeoutSecs` -> `navigationTimeoutSecs`
+* `handleFailedRequestFunction` -> `failedRequestHandler`
+
+We also renamed the crawling context interfaces, so they follow the same convention and are more meaningful:
+
+* `CheerioHandlePageInputs` -> `CheerioCrawlingContext`
+* `PlaywrightHandlePageFunction` -> `PlaywrightCrawlingContext`
+* `PuppeteerHandlePageFunction` -> `PuppeteerCrawlingContext`
+
+## Context aware helpers
+
+Some utilities previously available under `Apify.utils` namespace are now moved to the crawling context and are _context aware_. This means they have some parameters automatically filled in from the context, like the current `Request` instance or current `Page` object, or the `RequestQueue` bound to the crawler.
+
+### Enqueuing links
+
+One common helper that received more attention is the `enqueueLinks`. As mentioned above, it is context aware - we no longer need pass in the `requestQueue` or `page` arguments (or the cheerio handle `$`). In addition to that, it now offers 3 enqueuing strategies:
+
+* `EnqueueStrategy.All` (`'all'`): Matches any URLs found
+* `EnqueueStrategy.SameHostname` (`'same-hostname'`) Matches any URLs that have the same subdomain as the base URL (default)
+* `EnqueueStrategy.SameDomain` (`'same-domain'`) Matches any URLs that have the same domain name. For example, `https://wow.an.example.com` and `https://example.com` will both be matched for a base url of `https://example.com`.
+
+This means we can even call `enqueueLinks()` without any parameters. By default, it will go through all the links found on current page and filter only those targeting the same subdomain.
+
+Moreover, we can specify patterns the URL should match via globs:
+
+```ts
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks({
+ globs: ['https://crawlee.dev/*/*'],
+ // we can also use `regexps` and `pseudoUrls` keys here
+ });
+ },
+});
+```
+
+## Implicit `RequestQueue` instance
+
+All crawlers now have the `RequestQueue` instance automatically available via `crawler.getRequestQueue()` method. It will create the instance for you if it does not exist yet. This mean we no longer need to create the `RequestQueue` instance manually, and we can just use `crawler.addRequests()` method described underneath.
+
+> We can still create the `RequestQueue` explicitly, the `crawler.getRequestQueue()` method will respect that and return the instance provided via crawler options.
+
+## `crawler.addRequests()`
+
+We can now add multiple requests in batches. The newly added `addRequests` method will handle everything for us. It enqueues the first 1000 requests and resolves, while continuing with the rest in the background, again in a smaller 1000 items batches, so we don't fall into any API rate limits. This means the crawling will start almost immediately (within few seconds at most), something previously possible only with a combination of `RequestQueue` and `RequestList`.
+
+```ts
+// will resolve right after the initial batch of 1000 requests is added
+const result = await crawler.addRequests([/* many requests, can be even millions */]);
+
+// if we want to wait for all the requests to be added, we can await the `waitForAllRequestsToBeAdded` promise
+await result.waitForAllRequestsToBeAdded;
+```
+
+## Less verbose error logging
+
+Previously an error thrown from inside request handler resulted in full error object being logged. With Crawlee, we log only the error message as a warning as long as we know the request will be retried. If you want to enable verbose logging like in v2, use the `CRAWLEE_VERBOSE_LOG` env var.
+
+## `Request.label` shortcut
+
+Labeling requests used to work via the `Request.userData` object. With Crawlee, we can also use the `Request.label` shortcut. It is implemented as a `get/set` pair, using the value from `Request.userData`. The support for this shortcut is also added to the `enqueueLinks` options interface.
+
+```ts
+async requestHandler({ request, enqueueLinks }) {
+ if (request.label !== 'DETAIL') {
+ await enqueueLinks({
+ globs: ['...'],
+ label: 'DETAIL',
+ });
+ }
+}
+```
+
+## Removal of `requestAsBrowser`
+
+In v1 we replaced the underlying implementation of `requestAsBrowser` to be just a proxy over calling [`got-scraping`](https://github.com/apify/got-scraping) - our custom extension to `got` that tries to mimic the real browsers as much as possible. With v3, we are removing the `requestAsBrowser`, encouraging the use of [`got-scraping`](https://github.com/apify/got-scraping) directly.
+
+For easier migration, we also added `context.sendRequest()` helper that allows processing the context bound `Request` object through [`got-scraping`](https://github.com/apify/got-scraping):
+
+```ts
+const crawler = new BasicCrawler({
+ async requestHandler({ sendRequest, log }) {
+ // we can use the options parameter to override gotScraping options
+ const res = await sendRequest({ responseType: 'json' });
+ log.info('received body', res.body);
+ },
+});
+```
+
+### How to use `sendRequest()`?
+
+See [the Got Scraping guide](../guides/got_scraping.mdx).
+
+### Removed options
+
+The `useInsecureHttpParser` option has been removed. It's permanently set to `true` in order to better mimic browsers' behavior.
+
+Got Scraping automatically performs protocol negotiation, hence we removed the `useHttp2` option. It's set to `true` - 100% of browsers nowadays are capable of HTTP/2 requests. Oh, more and more of the web is using it too!
+
+### Renamed options
+
+In the `requestAsBrowser` approach, some of the options were named differently. Here's a list of renamed options:
+
+#### `payload`
+
+This options represents the body to send. It could be a `string` or a `Buffer`. However, there is no `payload` option anymore. You need to use `body` instead. Or, if you wish to send JSON, `json`. Here's an example:
+
+```ts
+// Before:
+await Apify.utils.requestAsBrowser({ …, payload: 'Hello, world!' });
+await Apify.utils.requestAsBrowser({ …, payload: Buffer.from('c0ffe', 'hex') });
+await Apify.utils.requestAsBrowser({ …, json: { hello: 'world' } });
+
+// After:
+await gotScraping({ …, body: 'Hello, world!' });
+await gotScraping({ …, body: Buffer.from('c0ffe', 'hex') });
+await gotScraping({ …, json: { hello: 'world' } });
+```
+
+#### `ignoreSslErrors`
+
+It has been renamed to `https.rejectUnauthorized`. By default, it's set to `false` for convenience. However, if you want to make sure the connection is secure, you can do the following:
+
+```ts
+// Before:
+await Apify.utils.requestAsBrowser({ …, ignoreSslErrors: false });
+
+// After:
+await gotScraping({ …, https: { rejectUnauthorized: true } });
+```
+
+Please note: the meanings are opposite! So we needed to invert the values as well.
+
+#### `header-generator` options
+
+`useMobileVersion`, `languageCode` and `countryCode` no longer exist. Instead, you need to use `headerGeneratorOptions` directly:
+
+```ts
+// Before:
+await Apify.utils.requestAsBrowser({
+ …,
+ useMobileVersion: true,
+ languageCode: 'en',
+ countryCode: 'US',
+});
+
+// After:
+await gotScraping({
+ …,
+ headerGeneratorOptions: {
+ devices: ['mobile'], // or ['desktop']
+ locales: ['en-US'],
+ },
+});
+```
+
+#### `timeoutSecs`
+
+In order to set a timeout, use `timeout.request` (which is **milliseconds** now).
+
+```ts
+// Before:
+await Apify.utils.requestAsBrowser({
+ …,
+ timeoutSecs: 30,
+});
+
+// After:
+await gotScraping({
+ …,
+ timeout: {
+ request: 30 * 1000,
+ },
+});
+```
+
+#### `throwOnHttpErrors`
+
+`throwOnHttpErrors` → `throwHttpErrors`. This options throws on unsuccessful HTTP status codes, for example `404`. By default, it's set to `false`.
+
+#### `decodeBody`
+
+`decodeBody` → `decompress`. This options decompresses the body. Defaults to `true` - please do not change this or websites will break (unless you know what you're doing!).
+
+#### `abortFunction`
+
+This function used to make the promise throw on specific responses, if it returned `true`. However, it wasn't that useful.
+
+You probably want to cancel the request instead, which you can do in the following way:
+
+```ts
+const promise = gotScraping(…);
+
+promise.on('request', request => {
+ // Please note this is not a Got Request instance, but a ClientRequest one.
+ // https://nodejs.org/api/http.html#class-httpclientrequest
+
+ if (request.protocol !== 'https:') {
+ // Unsecure request, abort.
+ promise.cancel();
+
+ // If you set `isStream` to `true`, please use `stream.destroy()` instead.
+ }
+});
+
+const response = await promise;
+```
+
+## Removal of browser pool plugin mixing
+
+Previously, you were able to have a browser pool that would mix Puppeteer and Playwright plugins (or even your own custom plugins if you've built any). As of this version, that is no longer allowed, and creating such a browser pool will cause an error to be thrown (it's expected that all plugins that will be used are of the same type).
+
+:::info Confused?
+
+As an example, this change disallows a pool to mix Puppeteer with Playwright. You can still create pools that use multiple Playwright plugins, each with a different launcher if you want!
+
+:::
+
+## Handling requests outside of browser
+
+One small feature worth mentioning is the ability to handle requests with browser crawlers outside the browser. To do that, we can use a combination of `Request.skipNavigation` and `context.sendRequest()`.
+
+Take a look at how to achieve this by checking out the [Skipping navigation for certain requests](../examples/skip-navigation) example!
+
+## Logging
+
+Crawlee exports the default `log` instance directly as a named export. We also have a scoped `log` instance provided in the crawling context - this one will log messages prefixed with the crawler name and should be preferred for logging inside the request handler.
+
+```ts
+const crawler = new CheerioCrawler({
+ async requestHandler({ log, request }) {
+ log.info(`Opened ${request.loadedUrl}`);
+ },
+});
+```
+
+## Auto-saved crawler state
+
+Every crawler instance now has `useState()` method that will return a state object we can use. It will be automatically saved when `persistState` event occurs. The value is cached, so we can freely call this method multiple times and get the exact same reference. No need to worry about saving the value either, as it will happen automatically.
+
+```ts
+const crawler = new CheerioCrawler({
+ async requestHandler({ crawler }) {
+ const state = await crawler.useState({ foo: [] as number[] });
+ // just change the value, no need to care about saving it
+ state.foo.push(123);
+ },
+});
+```
+
+## Apify SDK
+
+The Apify platform helpers can be now found in the Apify SDK (`apify` NPM package). It exports the `Actor` class that offers following static helpers:
+
+* `ApifyClient` shortcuts: `addWebhook()`, `call()`, `callTask()`, `metamorph()`
+* helpers for running on Apify platform: `init()`, `exit()`, `fail()`, `main()`, `isAtHome()`, `createProxyConfiguration()`
+* storage support: `getInput()`, `getValue()`, `openDataset()`, `openKeyValueStore()`, `openRequestQueue()`, `pushData()`, `setValue()`
+* events support: `on()`, `off()`
+* other utilities: `getEnv()`, `newClient()`, `reboot()`
+
+`Actor.main` is now just a syntax sugar around calling `Actor.init()` at the beginning and `Actor.exit()` at the end (plus wrapping the user function in try/catch block). All those methods are async and should be awaited - with node 16 we can use the top level await for that. In other words, following is equivalent:
+
+```ts
+import { Actor } from 'apify';
+
+await Actor.init();
+// your code
+await Actor.exit('Crawling finished!');
+```
+
+```ts
+import { Actor } from 'apify';
+
+await Actor.main(async () => {
+ // your code
+}, { statusMessage: 'Crawling finished!' });
+```
+
+`Actor.init()` will conditionally set the storage implementation of Crawlee to the `ApifyClient` when running on the Apify platform, or keep the default (memory storage) implementation otherwise. It will also subscribe to the websocket events (or mimic them locally). `Actor.exit()` will handle the tear down and calls `process.exit()` to ensure our process won't hang indefinitely for some reason.
+
+### Events
+
+Apify SDK (v2) exports `Apify.events`, which is an `EventEmitter` instance. With Crawlee, the events are managed by <ApiLink to="core/class/EventManager">`EventManager`</ApiLink> class instead. We can either access it via `Actor.eventManager` getter, or use `Actor.on` and `Actor.off` shortcuts instead.
+
+```diff
+-Apify.events.on(...);
++Actor.on(...);
+```
+
+> We can also get the <ApiLink to="core/class/EventManager">`EventManager`</ApiLink> instance via `Configuration.getEventManager()`.
+
+In addition to the existing events, we now have an `exit` event fired when calling `Actor.exit()` (which is called at the end of `Actor.main()`). This event allows you to gracefully shut down any resources when `Actor.exit` is called.
+
+## Smaller/internal breaking changes
+
+* `Apify.call()` is now just a shortcut for running `ApifyClient.actor(actorId).call(input, options)`, while also taking the token inside env vars into account
+* `Apify.callTask()` is now just a shortcut for running `ApifyClient.task(taskId).call(input, options)`, while also taking the token inside env vars into account
+* `Apify.metamorph()` is now just a shortcut for running `ApifyClient.task(taskId).metamorph(input, options)`, while also taking the ACTOR_RUN_ID inside env vars into account
+* `Apify.waitForRunToFinish()` has been removed, use `ApifyClient.waitForFinish()` instead
+* `Actor.main/init` purges the storage by default
+* remove `purgeLocalStorage` helper, move purging to the storage class directly
+ * `StorageClient` interface now has optional `purge` method
+ * purging happens automatically via `Actor.init()` (you can opt out via `purge: false` in the options of `init/main` methods)
+* `QueueOperationInfo.request` is no longer available
+* `Request.handledAt` is now string date in ISO format
+* `Request.inProgress` and `Request.reclaimed` are now `Set`s instead of POJOs
+* `injectUnderscore` from puppeteer utils has been removed
+* `APIFY_MEMORY_MBYTES` is no longer taken into account, use `CRAWLEE_AVAILABLE_MEMORY_RATIO` instead
+* some `AutoscaledPool` options are no longer available:
+ * `cpuSnapshotIntervalSecs` and `memorySnapshotIntervalSecs` has been replaced with top level `systemInfoIntervalMillis` configuration
+ * `maxUsedCpuRatio` has been moved to the top level configuration
+* `ProxyConfiguration.newUrlFunction` can be async. `.newUrl()` and `.newProxyInfo()` now return promises.
+* `prepareRequestFunction` and `postResponseFunction` options are removed, use navigation hooks instead
+* `gotoFunction` and `gotoTimeoutSecs` are removed
+* removed compatibility fix for old/broken request queues with null `Request` props
+* `fingerprintsOptions` renamed to `fingerprintOptions` (`fingerprints` -> `fingerprint`).
+* `fingerprintOptions` now accept `useFingerprintCache` and `fingerprintCacheSize` (instead of `useFingerprintPerProxyCache` and `fingerprintPerProxyCacheSize`, which are now no longer available). This is because the cached fingerprints are no longer connected to proxy URLs but to sessions.
diff --git a/website/versioned_sidebars/version-3.7-sidebars.json b/website/versioned_sidebars/version-3.7-sidebars.json
new file mode 100644
index 000000000..2c9df13ea
--- /dev/null
+++ b/website/versioned_sidebars/version-3.7-sidebars.json
@@ -0,0 +1,140 @@
+{
+ "docs": [
+ "quick-start/quick-start",
+ {
+ "type": "category",
+ "label": "Introduction",
+ "collapsed": false,
+ "link": {
+ "type": "doc",
+ "id": "introduction/introduction"
+ },
+ "items": [
+ "introduction/setting-up",
+ "introduction/first-crawler",
+ "introduction/adding-urls",
+ "introduction/real-world-project",
+ "introduction/crawling",
+ "introduction/scraping",
+ "introduction/saving-data",
+ "introduction/refactoring",
+ "introduction/deployment"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Guides",
+ "link": {
+ "type": "generated-index",
+ "title": "Guides",
+ "slug": "/guides",
+ "keywords": [
+ "guides"
+ ]
+ },
+ "items": [
+ "guides/request-storage",
+ "guides/result-storage",
+ "guides/configuration",
+ "guides/cheerio-crawler-guide",
+ "guides/javascript-rendering",
+ "guides/proxy-management",
+ "guides/session-management",
+ "guides/scaling-crawlers",
+ "guides/avoid-blocking",
+ "guides/jsdom-crawler-guide",
+ "guides/got-scraping",
+ "guides/typescript-project",
+ "guides/docker-images"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers.",
+ "slug": "/deployment"
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Examples",
+ "link": {
+ "type": "generated-index",
+ "title": "Examples",
+ "slug": "/examples",
+ "keywords": [
+ "examples"
+ ]
+ },
+ "items": [
+ {
+ "type": "autogenerated",
+ "dirName": "examples"
+ }
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Experiments",
+ "link": {
+ "type": "generated-index",
+ "title": "Experiments",
+ "slug": "/experiments",
+ "keywords": [
+ "experiments",
+ "experimental-features"
+ ]
+ },
+ "items": [
+ {
+ "type": "autogenerated",
+ "dirName": "experiments"
+ }
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Upgrading",
+ "link": {
+ "type": "generated-index",
+ "title": "Upgrading",
+ "slug": "/upgrading",
+ "keywords": [
+ "upgrading"
+ ]
+ },
+ "items": [
+ {
+ "type": "autogenerated",
+ "dirName": "upgrading"
+ }
+ ]
+ }
+ ]
+}
diff --git a/website/versions.json b/website/versions.json
index a473aab82..3e9c17b84 100644
--- a/website/versions.json
+++ b/website/versions.json
@@ -1,4 +1,5 @@
[
+ "3.7",
"3.6",
"3.5",
"3.4",