Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: website design uplift (#2872)
Daniel Wébr2e5899220939
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1568248a9..8d3ea85e0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1256,7 +1256,7 @@ const crawler = new BasicCrawler({#### How to use `sendRequest()`?-See [the Got Scraping guide](https://crawlee.dev/docs/guides/got-scraping).+See [the Got Scraping guide](https://crawlee.dev/js/docs/guides/got-scraping).#### Removed options@@ -1381,7 +1381,7 @@ Previously, you were able to have a browser pool that would mix Puppeteer and PlOne small feature worth mentioning is the ability to handle requests with browser crawlers outside the browser. To do that, we can use a combination of `Request.skipNavigation` and `context.sendRequest()`.-Take a look at how to achieve this by checking out the [Skipping navigation for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example!+Take a look at how to achieve this by checking out the [Skipping navigation for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example!### Logging@@ -1441,14 +1441,14 @@ await Actor.main(async () => {#### Events-Apify SDK (v2) exports `Apify.events`, which is an `EventEmitter` instance. With Crawlee, the events are managed by [`EventManager`](https://crawlee.dev/api/core/class/EventManager) class instead. We can either access it via `Actor.eventManager` getter, or use `Actor.on` and `Actor.off` shortcuts instead.+Apify SDK (v2) exports `Apify.events`, which is an `EventEmitter` instance. With Crawlee, the events are managed by [`EventManager`](https://crawlee.dev/js/api/core/class/EventManager) class instead. We can either access it via `Actor.eventManager` getter, or use `Actor.on` and `Actor.off` shortcuts instead.```diff-Apify.events.on(...);+Actor.on(...);```-> We can also get the [`EventManager`](https://crawlee.dev/api/core/class/EventManager) instance via `Configuration.getEventManager()`.+> We can also get the [`EventManager`](https://crawlee.dev/js/api/core/class/EventManager) instance via `Configuration.getEventManager()`.In addition to the existing events, we now have an `exit` event fired when calling `Actor.exit()` (which is called at the end of `Actor.main()`). This event allows you to gracefully shut down any resources when `Actor.exit` is called.diff --git a/MIGRATIONS.md b/MIGRATIONS.mdindex 70435b7e0..dc14ed265 100644--- a/MIGRATIONS.md+++ b/MIGRATIONS.md@@ -1,5 +1,5 @@# Migration from 2.x.x to 3.0.0-Check the v3 [upgrading guide](https://crawlee.dev/docs/upgrading/upgrading-to-v3).+Check the v3 [upgrading guide](https://crawlee.dev/js/docs/upgrading/upgrading-to-v3).# Migration from 1.x.x to 2.0.0There should be no changes needed apart from upgrading your Node.js version to >= 15.10. If you encounter issues with `cheerio`, [read their CHANGELOG](https://github.com/cheeriojs/cheerio/releases). We bumped it from `rc.3` to `rc.10`.diff --git a/README.md b/README.mdindex 72b7bcd73..c91beed7b 100644--- a/README.md+++ b/README.md@@ -28,7 +28,7 @@ Crawlee is available as the [`crawlee`](https://www.npmjs.com/package/crawlee) N## Installation-We recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.+We recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.> Crawlee requires **Node.js 16 or higher**.@@ -78,7 +78,7 @@ const crawler = new PlaywrightCrawler({await crawler.run(['https://crawlee.dev']);```-By default, Crawlee stores data to `./storage` in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).+By default, Crawlee stores data to `./storage` in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).### Installing pre-release versionsdiff --git a/docs/examples/cheerio_crawler.ts b/docs/examples/cheerio_crawler.tsindex 7a34547d1..04f41981b 100644--- a/docs/examples/cheerio_crawler.ts+++ b/docs/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/docs/examples/http_crawler.ts b/docs/examples/http_crawler.tsindex ef5492340..add990c59 100644--- a/docs/examples/http_crawler.ts+++ b/docs/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/docs/examples/jsdom_crawler.ts b/docs/examples/jsdom_crawler.tsindex f1760044f..6db02c4ef 100644--- a/docs/examples/jsdom_crawler.ts+++ b/docs/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/docs/experiments/request_locking.mdx b/docs/experiments/request_locking.mdxindex 224b59ec7..46ad112f4 100644--- a/docs/experiments/request_locking.mdx+++ b/docs/experiments/request_locking.mdx@@ -82,8 +82,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/docs/guides/docker_browser_js.txt b/docs/guides/docker_browser_js.txtindex 6e804b93a..76848bac3 100644--- a/docs/guides/docker_browser_js.txt+++ b/docs/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20diff --git a/docs/guides/docker_browser_ts.txt b/docs/guides/docker_browser_ts.txtindex 1fe6784a4..513bccfc1 100644--- a/docs/guides/docker_browser_ts.txt+++ b/docs/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20 AS builderdiff --git a/docs/guides/docker_node_js.txt b/docs/guides/docker_node_js.txtindex 4c8d11fc3..50cecc043 100644--- a/docs/guides/docker_node_js.txt+++ b/docs/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20diff --git a/docs/guides/docker_node_ts.txt b/docs/guides/docker_node_ts.txtindex 995a3d815..30d85ce51 100644--- a/docs/guides/docker_node_ts.txt+++ b/docs/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20 AS builderdiff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdxindex 279a504c9..8da5489b5 100644--- a/docs/guides/request_storage.mdx+++ b/docs/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/docs/guides/running-in-web-server/running-in-web-server.mdx b/docs/guides/running-in-web-server/running-in-web-server.mdxindex 39c347510..cee0f83a9 100644--- a/docs/guides/running-in-web-server/running-in-web-server.mdx+++ b/docs/guides/running-in-web-server/running-in-web-server.mdx@@ -10,7 +10,7 @@ import ApiLink from '@site/src/components/ApiLink';import WebServerSource from '!!raw-loader!./web-server.mjs';-Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.+Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.We will build a simple HTTP server that receives a page URL and returns the page title in the response. We will base this guide on the approach used in [Apify's Super Scraper API repository](https://github.com/apify/super-scraper) which maps incoming HTTP requests to Crawlee <ApiLink to="core/class/Request">Request</ApiLink>.diff --git a/docs/introduction/01-setting-up.mdx b/docs/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/docs/introduction/01-setting-up.mdx+++ b/docs/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/docs/introduction/03-adding-urls.mdx+++ b/docs/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/docs/quick-start/quick_start_cheerio.txt b/docs/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/docs/quick-start/quick_start_cheerio.txt+++ b/docs/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading_v3.mdindex aa62f6c08..39b6091c9 100644--- a/docs/upgrading/upgrading_v3.md+++ b/docs/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/packages/basic-crawler/README.md b/packages/basic-crawler/README.mdindex 032df77c5..e2a789779 100644--- a/packages/basic-crawler/README.md+++ b/packages/basic-crawler/README.md@@ -3,15 +3,15 @@Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.`BasicCrawler` is a low-level tool that requires the user to implement the page download and data extraction functionality themselves.-If we want a crawler that already facilitates this functionality, we should consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler).+If we want a crawler that already facilitates this functionality, we should consider using [CheerioCrawler](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), [PuppeteerCrawler](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler).-`BasicCrawler` invokes the user-provided [`requestHandler`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) for each [Request](https://crawlee.dev/api/core/class/Request) object, which represents a single URL to crawl. The [Request](https://crawlee.dev/api/core/class/Request) objects are fed from the [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/api/basic-crawler/class/BasicCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/api/basic-crawler/class/BasicCrawler#run) function is provided.+`BasicCrawler` invokes the user-provided [`requestHandler`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#requestHandler) for each [Request](https://crawlee.dev/js/api/core/class/Request) object, which represents a single URL to crawl. The [Request](https://crawlee.dev/js/api/core/class/Request) objects are fed from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/js/api/basic-crawler/class/BasicCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/js/api/basic-crawler/class/BasicCrawler#run) function is provided.-If both [`requestList`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [`requestList`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes if there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes if there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#autoscaledPoolOptions) parameter of the `BasicCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) constructor are available directly in the `BasicCrawler` constructor.+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#autoscaledPoolOptions) parameter of the `BasicCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) constructor are available directly in the `BasicCrawler` constructor.## Example usagediff --git a/packages/basic-crawler/src/internals/basic-crawler.ts b/packages/basic-crawler/src/internals/basic-crawler.tsindex 79439857c..46a3e4708 100644--- a/packages/basic-crawler/src/internals/basic-crawler.ts+++ b/packages/basic-crawler/src/internals/basic-crawler.ts@@ -74,7 +74,7 @@ export interface BasicCrawlingContext<UserData extends Dictionary = Dictionary>* Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions* and override settings of the enqueued {@apilink Request} objects.*- * Check out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example+ * Check out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example* for more details regarding its usage.** **Example usage**diff --git a/packages/browser-crawler/README.md b/packages/browser-crawler/README.mdindex e9d635d3f..904d7a553 100644--- a/packages/browser-crawler/README.md+++ b/packages/browser-crawler/README.md@@ -2,17 +2,17 @@Provides a simple framework for parallel crawling of web pages using headless browsers with [Puppeteer](https://github.com/puppeteer/puppeteer) and [Playwright](https://github.com/microsoft/playwright). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `BrowserCrawler` uses headless (or even headful) browsers to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, we should consider using the [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.+Since `BrowserCrawler` uses headless (or even headful) browsers to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, we should consider using the [CheerioCrawler](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.-The source URLs are represented by the [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from the [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/api/browser-crawler/class/BrowserCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/api/browser-crawler/class/BrowserCrawler#run) function is provided.+The source URLs are represented by the [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [`requestList`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) or [`requestQueue`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) constructor options, respectively. If neither `requestList` nor `requestQueue` options are provided, the crawler will open the default request queue either when the [`crawler.addRequests()`](https://crawlee.dev/js/api/browser-crawler/class/BrowserCrawler#addRequests) function is called, or if `requests` parameter (representing the initial requests) of the [`crawler.run()`](https://crawlee.dev/js/api/browser-crawler/class/BrowserCrawler#run) function is provided.-If both [`requestList`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [`requestList`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#requestList) and [`requestQueue`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#requestQueue) options are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to the [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.-`BrowserCrawler` opens a new browser page (i.e. tab or window) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl and then calls the function provided by user as the [`requestHandler`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#requestHandler) option.+`BrowserCrawler` opens a new browser page (i.e. tab or window) for each [Request](https://crawlee.dev/js/api/core/class/Request) object to crawl and then calls the function provided by user as the [`requestHandler`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#requestHandler) option.-New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class.-All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#autoscaledPoolOptions) parameter of the `BrowserCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) constructor are available directly in the `BrowserCrawler` constructor.+New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class.+All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the [`autoscaledPoolOptions`](https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerOptions#autoscaledPoolOptions) parameter of the `BrowserCrawler` constructor. For user convenience, the [`minConcurrency`](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions#minConcurrency) and [`maxConcurrency`](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions#maxConcurrency) options of the underlying [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) constructor are available directly in the `BrowserCrawler` constructor.-> *NOTE:* the pool of browser instances is internally managed by the [BrowserPool](https://crawlee.dev/api/browser-pool/class/BrowserPool) class.+> *NOTE:* the pool of browser instances is internally managed by the [BrowserPool](https://crawlee.dev/js/api/browser-pool/class/BrowserPool) class.diff --git a/packages/browser-crawler/src/internals/browser-launcher.ts b/packages/browser-crawler/src/internals/browser-launcher.tsindex 87e6366e9..0fd487fa9 100644--- a/packages/browser-crawler/src/internals/browser-launcher.ts+++ b/packages/browser-crawler/src/internals/browser-launcher.ts@@ -77,7 +77,7 @@ export interface BrowserLaunchContext<TOptions, Launcher> extends BrowserPluginO* import { firefox } from 'playwright';* ```*- * For more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox).+ * For more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox).*/launcher?: Launcher;}diff --git a/packages/browser-pool/README.md b/packages/browser-pool/README.mdindex 6fa1eca17..13b78fcd3 100644--- a/packages/browser-pool/README.md+++ b/packages/browser-pool/README.md@@ -293,4 +293,4 @@ With Browser Pool, browsers are not closed, but retired. A retired browser will> Fingerprints are enabled by default since v3.-Changing browser fingerprints is beneficial for avoiding getting blocked and simulating real user browsers. With Browser Pool, you can do this otherwise complicated technique by enabling the `useFingerprints` option. The fingerprints are by default tied to the respective proxy urls to not use the same unique fingerprint from various IP addresses. You can disable this behavior in the [`fingerprintOptions`](/api/browser-pool/interface/FingerprintOptions). In the `fingerprintOptions`, You can also control which fingerprints are generated. You can control parameters as browser, operating system, and browser versions.+Changing browser fingerprints is beneficial for avoiding getting blocked and simulating real user browsers. With Browser Pool, you can do this otherwise complicated technique by enabling the `useFingerprints` option. The fingerprints are by default tied to the respective proxy urls to not use the same unique fingerprint from various IP addresses. You can disable this behavior in the [`fingerprintOptions`](/js/api/browser-pool/interface/FingerprintOptions). In the `fingerprintOptions`, You can also control which fingerprints are generated. You can control parameters as browser, operating system, and browser versions.diff --git a/packages/cheerio-crawler/README.md b/packages/cheerio-crawler/README.mdindex c16a4c601..d98911284 100644--- a/packages/cheerio-crawler/README.md+++ b/packages/cheerio-crawler/README.md@@ -2,15 +2,15 @@Provides a framework for the parallel crawling of web pages using plain HTTP requests and [cheerio](https://www.npmjs.com/package/cheerio) HTML parser. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `CheerioCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.+Since `CheerioCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.-`CheerioCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio) and then invokes the user-provided [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler) to extract page data using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.+`CheerioCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [Cheerio](https://www.npmjs.com/package/cheerio) and then invokes the user-provided [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler) to extract page data using a [jQuery](https://jquery.com/)-like interface to the parsed HTML DOM.-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [CheerioCrawlerOptions.requestList](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) or [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [CheerioCrawlerOptions.requestList](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) or [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) constructor options, respectively.-If both [CheerioCrawlerOptions.requestList](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) and [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [CheerioCrawlerOptions.requestList](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestList) and [CheerioCrawlerOptions.requestQueue](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.We can use the `preNavigationHooks` to adjust `gotOptions`:@@ -22,9 +22,9 @@ preNavigationHooks: []```-By default, `CheerioCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [CheerioCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler).+By default, `CheerioCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [CheerioCrawlerOptions.additionalMimeTypes](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [CheerioCrawlerOptions.requestHandler](https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler).-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `CheerioCrawler` constructor.+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `CheerioCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) options are available directly in the `CheerioCrawler` constructor.## Example usagediff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.mdindex 50b642ef4..f1527ca3d 100644--- a/packages/core/CHANGELOG.md+++ b/packages/core/CHANGELOG.md@@ -1043,7 +1043,7 @@ const crawler = new BasicCrawler({#### How to use `sendRequest()`?-See [the Got Scraping guide](https://crawlee.dev/docs/guides/got-scraping).+See [the Got Scraping guide](https://crawlee.dev/js/docs/guides/got-scraping).#### Removed options@@ -1168,7 +1168,7 @@ Previously, you were able to have a browser pool that would mix Puppeteer and PlOne small feature worth mentioning is the ability to handle requests with browser crawlers outside the browser. To do that, we can use a combination of `Request.skipNavigation` and `context.sendRequest()`.-Take a look at how to achieve this by checking out the [Skipping navigation for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example!+Take a look at how to achieve this by checking out the [Skipping navigation for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example!### Logging@@ -1228,14 +1228,14 @@ await Actor.main(async () => {#### Events-Apify SDK (v2) exports `Apify.events`, which is an `EventEmitter` instance. With Crawlee, the events are managed by [`EventManager`](https://crawlee.dev/api/core/class/EventManager) class instead. We can either access it via `Actor.eventManager` getter, or use `Actor.on` and `Actor.off` shortcuts instead.+Apify SDK (v2) exports `Apify.events`, which is an `EventEmitter` instance. With Crawlee, the events are managed by [`EventManager`](https://crawlee.dev/js/api/core/class/EventManager) class instead. We can either access it via `Actor.eventManager` getter, or use `Actor.on` and `Actor.off` shortcuts instead.```diff-Apify.events.on(...);+Actor.on(...);```-> We can also get the [`EventManager`](https://crawlee.dev/api/core/class/EventManager) instance via `Configuration.getEventManager()`.+> We can also get the [`EventManager`](https://crawlee.dev/js/api/core/class/EventManager) instance via `Configuration.getEventManager()`.In addition to the existing events, we now have an `exit` event fired when calling `Actor.exit()` (which is called at the end of `Actor.main()`). This event allows you to gracefully shut down any resources when `Actor.exit` is called.diff --git a/packages/core/README.md b/packages/core/README.mdindex da3150e0f..167cf6112 100644--- a/packages/core/README.md+++ b/packages/core/README.md@@ -4,19 +4,19 @@ Core set of classes required for Crawlee.The [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/linkedom`](https://crawlee.dev/api/linkedom-crawler): exports `LinkeDOMCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/linkedom`](https://crawlee.dev/js/api/linkedom-crawler): exports `LinkeDOMCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)## Installing Crawleediff --git a/packages/core/src/crawlers/crawler_commons.ts b/packages/core/src/crawlers/crawler_commons.tsindex 5ea0e71e3..14a1d1f96 100644--- a/packages/core/src/crawlers/crawler_commons.ts+++ b/packages/core/src/crawlers/crawler_commons.ts@@ -60,7 +60,7 @@ export interface RestrictedCrawlingContext<UserData extends Dictionary = Diction* Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions* and override settings of the enqueued {@apilink Request} objects.*- * Check out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example+ * Check out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example* for more details regarding its usage.** **Example usage**@@ -119,7 +119,7 @@ export interface CrawlingContext<Crawler = unknown, UserData extends Dictionary* Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions* and override settings of the enqueued {@apilink Request} objects.*- * Check out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example+ * Check out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example* for more details regarding its usage.** **Example usage**@@ -147,11 +147,11 @@ export interface CrawlingContext<Crawler = unknown, UserData extends DictionarygetKeyValueStore: (idOrName?: string) => Promise<KeyValueStore>;/**- * Fires HTTP request via [`got-scraping`](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request+ * Fires HTTP request via [`got-scraping`](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request* options on the fly.** This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).- * Check the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for+ * Check the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for* more detailed explanation of how to do that.** ```tsdiff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/core/src/enqueue_links/enqueue_links.tsindex 5373dee69..881d1f595 100644--- a/packages/core/src/enqueue_links/enqueue_links.ts+++ b/packages/core/src/enqueue_links/enqueue_links.ts@@ -245,7 +245,7 @@ export async function enqueueLinks(throw new RangeError(['enqueueLinks() was called without the required options. You can only do that when you use the `crawlingContext.enqueueLinks()` method in request handlers.',- 'Check out our guide on how to use enqueueLinks() here: https://crawlee.dev/docs/examples/crawl-relative-links',+ 'Check out our guide on how to use enqueueLinks() here: https://crawlee.dev/js/docs/examples/crawl-relative-links',].join('\n'),);}diff --git a/packages/http-crawler/README.md b/packages/http-crawler/README.mdindex 55d0c43c1..05dd473c9 100644--- a/packages/http-crawler/README.md+++ b/packages/http-crawler/README.md@@ -2,13 +2,13 @@Provides a framework for the parallel crawling of web pages using plain HTTP requests. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-It is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser. **This crawler downloads each URL using a plain HTTP request and doesn't do any HTML parsing.**+It is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser. **This crawler downloads each URL using a plain HTTP request and doesn't do any HTML parsing.**-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [HttpCrawlerOptions.requestList](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestList) or [HttpCrawlerOptions.requestQueue](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [HttpCrawlerOptions.requestList](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestList) or [HttpCrawlerOptions.requestQueue](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestQueue) constructor options, respectively.-If both [HttpCrawlerOptions.requestList](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestList) and [HttpCrawlerOptions.requestQueue](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [HttpCrawlerOptions.requestList](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestList) and [HttpCrawlerOptions.requestQueue](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.We can use the `preNavigationHooks` to adjust `gotOptions`:@@ -20,9 +20,9 @@ preNavigationHooks: []```-By default, `HttpCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [HttpCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [HttpCrawlerOptions.requestHandler](https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler).+By default, `HttpCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [HttpCrawlerOptions.additionalMimeTypes](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [HttpCrawlerOptions.requestHandler](https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler).-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `HttpCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `HttpCrawler` constructor.+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `HttpCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) options are available directly in the `HttpCrawler` constructor.## Example usagediff --git a/packages/jsdom-crawler/README.md b/packages/jsdom-crawler/README.mdindex 3decfc6bf..afde8250d 100644--- a/packages/jsdom-crawler/README.md+++ b/packages/jsdom-crawler/README.md@@ -2,15 +2,15 @@Provides a framework for the parallel crawling of web pages using plain HTTP requests and [jsdom](https://www.npmjs.com/package/jsdom) DOM implementation. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `JSDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.+Since `JSDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.-`JSDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [JSDOM](https://www.npmjs.com/package/jsdom) and then invokes the user-provided [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.+`JSDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [JSDOM](https://www.npmjs.com/package/jsdom) and then invokes the user-provided [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [JSDOMCrawlerOptions.requestList](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) or [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [JSDOMCrawlerOptions.requestList](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) or [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) constructor options, respectively.-If both [JSDOMCrawlerOptions.requestList](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) and [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [JSDOMCrawlerOptions.requestList](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestList) and [JSDOMCrawlerOptions.requestQueue](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.We can use the `preNavigationHooks` to adjust `gotOptions`:@@ -22,9 +22,9 @@ preNavigationHooks: []```-By default, `JSDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [JSDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler).+By default, `JSDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [JSDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [JSDOMCrawlerOptions.requestHandler](https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler).-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `JSDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `JSDOMCrawler` constructor.+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `JSDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) options are available directly in the `JSDOMCrawler` constructor.## Example usagediff --git a/packages/linkedom-crawler/README.md b/packages/linkedom-crawler/README.mdindex b3b506cd2..014749c81 100644--- a/packages/linkedom-crawler/README.md+++ b/packages/linkedom-crawler/README.md@@ -4,15 +4,15 @@Provides a framework for the parallel crawling of web pages using plain HTTP requests and [linkedom](https://www.npmjs.com/package/linkedom) DOM implementation. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `LinkeDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.+Since `LinkeDOMCrawler` uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript to display the content, you might need to use [PuppeteerCrawler](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler) or [PlaywrightCrawler](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler) instead, because it loads the pages using full-featured headless Chrome browser.-`LinkeDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [LinkeDOM](https://www.npmjs.com/package/linkedom) and then invokes the user-provided [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.+`LinkeDOMCrawler` downloads each URL using a plain HTTP request, parses the HTML content using [LinkeDOM](https://www.npmjs.com/package/linkedom) and then invokes the user-provided [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler) to extract page data using the `window` object.-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) or [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) or [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) constructor options, respectively.-If both [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) and [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [LinkeDOMCrawlerOptions.requestList](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestList) and [LinkeDOMCrawlerOptions.requestQueue](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.We can use the `preNavigationHooks` to adjust `gotOptions`:@@ -24,9 +24,9 @@ preNavigationHooks: []```-By default, `LinkeDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [LinkeDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler).+By default, `LinkeDOMCrawler` only processes web pages with the `text/html` and `application/xhtml+xml` MIME content types (as reported by the `Content-Type` HTTP header), and skips pages with other content types. If you want the crawler to process other content types, use the [LinkeDOMCrawlerOptions.additionalMimeTypes](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#additionalMimeTypes) constructor option. Beware that the parsing behavior differs for HTML, XML, JSON and other types of content. For more details, see [LinkeDOMCrawlerOptions.requestHandler](https://crawlee.dev/js/api/linkedom-crawler/interface/LinkeDOMCrawlerOptions#requestHandler).-New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `LinkeDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) options are available directly in the `LinkeDOMCrawler` constructor.+New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the `autoscaledPoolOptions` parameter of the `LinkeDOMCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) options are available directly in the `LinkeDOMCrawler` constructor.## Example usagediff --git a/packages/playwright-crawler/README.md b/packages/playwright-crawler/README.mdindex 858ee14fd..85728e399 100644--- a/packages/playwright-crawler/README.md+++ b/packages/playwright-crawler/README.md@@ -2,17 +2,17 @@Provides a simple framework for parallel crawling of web pages using headless Chromium, Firefox and Webkit browsers with [Playwright](https://github.com/microsoft/playwright). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `Playwright` uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.+Since `Playwright` uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) or [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) or [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) constructor options, respectively.-If both [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) and [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [PlaywrightCrawlerOptions.requestList](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestList) and [PlaywrightCrawlerOptions.requestQueue](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestQueue) are used, the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.-`PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl and then calls the function provided by user as the [PlaywrightCrawlerOptions.requestHandler](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestHandler) option.+`PlaywrightCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/js/api/core/class/Request) object to crawl and then calls the function provided by user as the [PlaywrightCrawlerOptions.requestHandler](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#requestHandler) option.-New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [PlaywrightCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#autoscaledPoolOptions) parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPoolOptions](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions) are available directly in the `PlaywrightCrawler` constructor.+New pages are only opened when there is enough free CPU and memory available, using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class. All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the [PlaywrightCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/js/api/playwright-crawler/interface/PlaywrightCrawlerOptions#autoscaledPoolOptions) parameter of the `PlaywrightCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency` [AutoscaledPoolOptions](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions) are available directly in the `PlaywrightCrawler` constructor.Note that the pool of Playwright instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.diff --git a/packages/puppeteer-crawler/README.md b/packages/puppeteer-crawler/README.mdindex 6a5a3ff66..27b6b3e06 100644--- a/packages/puppeteer-crawler/README.md+++ b/packages/puppeteer-crawler/README.md@@ -2,24 +2,24 @@Provides a simple framework for parallel crawling of web pages using headless Chrome with [Puppeteer](https://github.com/puppeteer/puppeteer). The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.-Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.+Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using [CheerioCrawler](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), which downloads the pages using raw HTTP requests and is about 10x faster.-The source URLs are represented using [Request](https://crawlee.dev/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) instances provided by the [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) or [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) constructor options, respectively.+The source URLs are represented using [Request](https://crawlee.dev/js/api/core/class/Request) objects that are fed from [RequestList](https://crawlee.dev/js/api/core/class/RequestList) or [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) instances provided by the [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) or [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) constructor options, respectively.-If both [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) and [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) are used,-the instance first processes URLs from the [RequestList](https://crawlee.dev/api/core/class/RequestList) and automatically enqueues all of them-to [RequestQueue](https://crawlee.dev/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.+If both [PuppeteerCrawlerOptions.requestList](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestList) and [PuppeteerCrawlerOptions.requestQueue](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestQueue) are used,+the instance first processes URLs from the [RequestList](https://crawlee.dev/js/api/core/class/RequestList) and automatically enqueues all of them+to [RequestQueue](https://crawlee.dev/js/api/core/class/RequestQueue) before it starts their processing. This ensures that a single URL is not crawled multiple times.-The crawler finishes when there are no more [Request](https://crawlee.dev/api/core/class/Request) objects to crawl.+The crawler finishes when there are no more [Request](https://crawlee.dev/js/api/core/class/Request) objects to crawl.-`PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/api/core/class/Request) object to crawl-and then calls the function provided by user as the [PuppeteerCrawlerOptions.requestHandler](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestHandler) option.+`PuppeteerCrawler` opens a new Chrome page (i.e. tab) for each [Request](https://crawlee.dev/js/api/core/class/Request) object to crawl+and then calls the function provided by user as the [PuppeteerCrawlerOptions.requestHandler](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#requestHandler) option.New pages are only opened when there is enough free CPU and memory available,-using the functionality provided by the [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) class.-All [AutoscaledPool](https://crawlee.dev/api/core/class/AutoscaledPool) configuration options can be passed to the [PuppeteerCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#autoscaledPoolOptions)+using the functionality provided by the [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) class.+All [AutoscaledPool](https://crawlee.dev/js/api/core/class/AutoscaledPool) configuration options can be passed to the [PuppeteerCrawlerOptions.autoscaledPoolOptions](https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions#autoscaledPoolOptions)parameter of the `PuppeteerCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`-[AutoscaledPoolOptions](https://crawlee.dev/api/core/interface/AutoscaledPoolOptions) are available directly in the `PuppeteerCrawler` constructor.+[AutoscaledPoolOptions](https://crawlee.dev/js/api/core/interface/AutoscaledPoolOptions) are available directly in the `PuppeteerCrawler` constructor.Note that the pool of Puppeteer instances is internally managed by the [BrowserPool](https://github.com/apify/browser-pool) class.diff --git a/packages/templates/templates/camoufox-ts/README.md b/packages/templates/templates/camoufox-ts/README.mdindex 59c80989e..85d44ee6a 100644--- a/packages/templates/templates/camoufox-ts/README.md+++ b/packages/templates/templates/camoufox-ts/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with `PlaywrightCIf you're looking for examples or want to learn more visit:-- [Documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)-- [Examples](https://crawlee.dev/docs/examples/playwright-crawler)+- [Documentation](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/playwright-crawler)diff --git a/packages/templates/templates/cheerio-js/README.md b/packages/templates/templates/cheerio-js/README.mdindex b888e6acb..5eb70104f 100644--- a/packages/templates/templates/cheerio-js/README.md+++ b/packages/templates/templates/cheerio-js/README.md@@ -4,6 +4,6 @@ This template is a production ready boilerplate for developing with `CheerioCrawIf you're looking for examples or want to learn more visit:-- [Tutorial](https://crawlee.dev/docs/guides/cheerio-crawler-guide)-- [Documentation](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler)-- [Examples](https://crawlee.dev/docs/examples/cheerio-crawler)+- [Tutorial](https://crawlee.dev/js/docs/guides/cheerio-crawler-guide)+- [Documentation](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/cheerio-crawler)diff --git a/packages/templates/templates/cheerio-ts/README.md b/packages/templates/templates/cheerio-ts/README.mdindex 585998e21..4eb1998bc 100644--- a/packages/templates/templates/cheerio-ts/README.md+++ b/packages/templates/templates/cheerio-ts/README.md@@ -4,6 +4,6 @@ This template is a production ready boilerplate for developing with `CheerioCrawIf you're looking for examples or want to learn more visit:-- [Tutorial](https://crawlee.dev/docs/guides/cheerio-crawler-guide)-- [Documentation](https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler)-- [Examples](https://crawlee.dev/docs/examples/cheerio-crawler)+- [Tutorial](https://crawlee.dev/js/docs/guides/cheerio-crawler-guide)+- [Documentation](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/cheerio-crawler)diff --git a/packages/templates/templates/empty-js/README.md b/packages/templates/templates/empty-js/README.mdindex c4b93d152..10c8f0946 100644--- a/packages/templates/templates/empty-js/README.md+++ b/packages/templates/templates/empty-js/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with Crawlee. UseIf you're looking for examples or want to learn more visit:-- [Quick Start](https://crawlee.dev/docs/quick-start)-- [Examples](https://crawlee.dev/docs/examples)+- [Quick Start](https://crawlee.dev/js/docs/quick-start)+- [Examples](https://crawlee.dev/js/docs/examples)diff --git a/packages/templates/templates/empty-ts/README.md b/packages/templates/templates/empty-ts/README.mdindex 2257e5aaa..64db9c04c 100644--- a/packages/templates/templates/empty-ts/README.md+++ b/packages/templates/templates/empty-ts/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with Crawlee. UseIf you're looking for examples or want to learn more visit:-- [Quick Start](https://crawlee.dev/docs/quick-start)-- [Examples](https://crawlee.dev/docs/examples)+- [Quick Start](https://crawlee.dev/js/docs/quick-start)+- [Examples](https://crawlee.dev/js/docs/examples)diff --git a/packages/templates/templates/getting-started-js/README.md b/packages/templates/templates/getting-started-js/README.mdindex 6a404f0e3..5c8e78008 100644--- a/packages/templates/templates/getting-started-js/README.md+++ b/packages/templates/templates/getting-started-js/README.md@@ -4,6 +4,6 @@ This example uses `PlaywrightCrawler` to recursively crawl https://crawlee.dev uYou can find more examples and documentation at the following links:-- [Step-by-step tutorial](https://crawlee.dev/docs/introduction) for Crawlee-- `PlaywrightCrawler` [API documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)-- Other [examples](https://crawlee.dev/docs/examples/playwright-crawler)+- [Step-by-step tutorial](https://crawlee.dev/js/docs/introduction) for Crawlee+- `PlaywrightCrawler` [API documentation](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler)+- Other [examples](https://crawlee.dev/js/docs/examples/playwright-crawler)diff --git a/packages/templates/templates/getting-started-ts/README.md b/packages/templates/templates/getting-started-ts/README.mdindex 6a404f0e3..5c8e78008 100644--- a/packages/templates/templates/getting-started-ts/README.md+++ b/packages/templates/templates/getting-started-ts/README.md@@ -4,6 +4,6 @@ This example uses `PlaywrightCrawler` to recursively crawl https://crawlee.dev uYou can find more examples and documentation at the following links:-- [Step-by-step tutorial](https://crawlee.dev/docs/introduction) for Crawlee-- `PlaywrightCrawler` [API documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)-- Other [examples](https://crawlee.dev/docs/examples/playwright-crawler)+- [Step-by-step tutorial](https://crawlee.dev/js/docs/introduction) for Crawlee+- `PlaywrightCrawler` [API documentation](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler)+- Other [examples](https://crawlee.dev/js/docs/examples/playwright-crawler)diff --git a/packages/templates/templates/playwright-js/README.md b/packages/templates/templates/playwright-js/README.mdindex 5dfa84acf..7d4195089 100644--- a/packages/templates/templates/playwright-js/README.md+++ b/packages/templates/templates/playwright-js/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with `PlaywrightCIf you're looking for examples or want to learn more visit:-- [Documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)-- [Examples](https://crawlee.dev/docs/examples/playwright-crawler)+- [Documentation](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/playwright-crawler)diff --git a/packages/templates/templates/playwright-ts/README.md b/packages/templates/templates/playwright-ts/README.mdindex 9f3345c75..879986bc0 100644--- a/packages/templates/templates/playwright-ts/README.md+++ b/packages/templates/templates/playwright-ts/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with `PlaywrightCIf you're looking for examples or want to learn more visit:-- [Documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)-- [Examples](https://crawlee.dev/docs/examples/playwright-crawler)+- [Documentation](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/playwright-crawler)diff --git a/packages/templates/templates/puppeteer-js/README.md b/packages/templates/templates/puppeteer-js/README.mdindex 01c3e360a..fac6b99f9 100644--- a/packages/templates/templates/puppeteer-js/README.md+++ b/packages/templates/templates/puppeteer-js/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with `PuppeteerCrIf you're looking for examples or want to learn more visit:-- [Documentation](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler)-- [Examples](https://crawlee.dev/docs/examples/puppeteer-crawler)+- [Documentation](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/puppeteer-crawler)diff --git a/packages/templates/templates/puppeteer-ts/README.md b/packages/templates/templates/puppeteer-ts/README.mdindex 68e0895ea..2c1babbcb 100644--- a/packages/templates/templates/puppeteer-ts/README.md+++ b/packages/templates/templates/puppeteer-ts/README.md@@ -4,5 +4,5 @@ This template is a production ready boilerplate for developing with `PuppeteerCrIf you're looking for examples or want to learn more visit:-- [Documentation](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler)-- [Examples](https://crawlee.dev/docs/examples/puppeteer-crawler)+- [Documentation](https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler)+- [Examples](https://crawlee.dev/js/docs/examples/puppeteer-crawler)diff --git a/packages/utils/src/internals/robots.ts b/packages/utils/src/internals/robots.tsindex 98d78e278..b3db16b01 100644--- a/packages/utils/src/internals/robots.ts+++ b/packages/utils/src/internals/robots.ts@@ -14,7 +14,7 @@ let HTTPError: typeof HTTPErrorClass;* **Example usage:*** ```javascript* // Load the robots.txt file- * const robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');+ * const robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');** // Check if a URL should be crawled according to robots.txt* const url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';diff --git a/website/babel.config.js b/website/babel.config.jsdeleted file mode 100644index 0adade1fb..000000000--- a/website/babel.config.js+++ /dev/null@@ -1,3 +0,0 @@-module.exports = {- presets: [require.resolve('@docusaurus/core/lib/babel/preset')],-};diff --git a/website/blog/2024/02-22-launching-crawlee-blog/index.md b/website/blog/2024/02-22-launching-crawlee-blog/index.mdindex 0806c5ab0..d3f742b68 100644--- a/website/blog/2024/02-22-launching-crawlee-blog/index.md+++ b/website/blog/2024/02-22-launching-crawlee-blog/index.md@@ -20,11 +20,11 @@ Since then, our dev community has grown exponentially. I’m proud to tell you tCrawlee has progressively evolved with the introduction of key features to enhance web scraping and automation:-- [v3.1](https://github.com/apify/crawlee/releases/tag/v3.1.0) added an [error tracker](https://crawlee.dev/api/core/class/ErrorTracker) for analyzing and summarizing failed requests.+- [v3.1](https://github.com/apify/crawlee/releases/tag/v3.1.0) added an [error tracker](https://crawlee.dev/js/api/core/class/ErrorTracker) for analyzing and summarizing failed requests.- The [v3.3](https://github.com/apify/crawlee/releases/tag/v3.3.0) update brought an `exclude` option to the `enqueueLinks` helper and integrated status messages. This improved usability on the Apify platform with automatic summary updates in the console UI.-- [v3.4](https://github.com/apify/crawlee/releases/tag/v3.4.0) introduced the [`linkedom` crawler](https://crawlee.dev/api/linkedom-crawler), offering a new parsing option.+- [v3.4](https://github.com/apify/crawlee/releases/tag/v3.4.0) introduced the [`linkedom` crawler](https://crawlee.dev/js/api/linkedom-crawler), offering a new parsing option.- The [v3.5](https://github.com/apify/crawlee/releases/tag/v3.5.0) update optimized link enqueuing for efficiency.-- [v3.6](https://github.com/apify/crawlee/releases/tag/v3.6.0) launched experimental support for a [new request queue API](https://crawlee.dev/docs/experiments/experiments-request-locking), enabling parallel execution and improved scalability for multiple scrapers working concurrently.+- [v3.6](https://github.com/apify/crawlee/releases/tag/v3.6.0) launched experimental support for a [new request queue API](https://crawlee.dev/js/docs/experiments/experiments-request-locking), enabling parallel execution and improved scalability for multiple scrapers working concurrently.All of this marked significant strides in making web scraping more efficient and robust.@@ -32,7 +32,7 @@ All of this marked significant strides in making web scraping more efficient andThe Crawlee team is actively developing an adaptive crawling feature to revolutionize how Crawlee interacts with and navigates through websites.-We just launched [v3.8](https://github.com/apify/crawlee/releases/tag/v3.8.0) with experimental support for the new [adaptive crawler type](https://crawlee.dev/api/playwright-crawler/class/AdaptivePlaywrightCrawler).+We just launched [v3.8](https://github.com/apify/crawlee/releases/tag/v3.8.0) with experimental support for the new [adaptive crawler type](https://crawlee.dev/js/api/playwright-crawler/class/AdaptivePlaywrightCrawler).## Support us on GitHub.diff --git a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.mdindex 6e67f2e04..3d857a337 100644--- a/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md+++ b/website/blog/2024/03-27-how-to-scrape-amazon-using-typescript-cheerio-and-crawlee/index.md@@ -424,7 +424,7 @@ While Crawlee's browser-like user-agent headers prevent blocking to a certain ex### Using proxies-The use of proxies marks another significant tactic in evading blocking. You'll be pleased to know that Crawlee excels in this domain, supporting both [custom proxies](https://crawlee.dev/docs/guides/proxy-management) and [Apify proxies](https://apify.com/proxy).+The use of proxies marks another significant tactic in evading blocking. You'll be pleased to know that Crawlee excels in this domain, supporting both [custom proxies](https://crawlee.dev/js/docs/guides/proxy-management) and [Apify proxies](https://apify.com/proxy).Here's an example of how to use Apify's [residential proxies](https://docs.apify.com/platform/proxy/residential-proxy), which are highly effective in preventing blocking:@@ -444,7 +444,7 @@ const crawler = new CheerioCrawler({ requestHandler, proxyConfiguration });### Using headless browsers to scrape Amazon-For more advanced scraping, you can use a headless browser like [Playwright](https://crawlee.dev/docs/examples/playwright-crawler) to scrape Amazon. This method is more effective in preventing blocking and can handle websites with complex JavaScript interactions.+For more advanced scraping, you can use a headless browser like [Playwright](https://crawlee.dev/js/docs/examples/playwright-crawler) to scrape Amazon. This method is more effective in preventing blocking and can handle websites with complex JavaScript interactions.To use Playwright with Crawlee, we can replace the `CheerioCrawler` with `PlaywrightCrawler`:diff --git a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md b/website/blog/2024/04-23-scrapy-vs-crawlee/index.mdindex 107c66853..820439949 100644--- a/website/blog/2024/04-23-scrapy-vs-crawlee/index.md+++ b/website/blog/2024/04-23-scrapy-vs-crawlee/index.md@@ -126,7 +126,7 @@ Autoscaling refers to the capability of a library to automatically adjusting theScrapy does not have built-in autoscaling capabilities, but it can be done using external services like [Scrapyd](https://scrapyd.readthedocs.io/en/latest/) or deployed in a distributed manner with Scrapy Cluster.-Crawlee has [built-in autoscaling](https://crawlee.dev/api/core/class/AutoscaledPool) with `AutoscaledPool`. It increases the number of requests that are processed concurrently within one crawler.+Crawlee has [built-in autoscaling](https://crawlee.dev/js/api/core/class/AutoscaledPool) with `AutoscaledPool`. It increases the number of requests that are processed concurrently within one crawler.### Queue management@@ -150,7 +150,7 @@ Scrapy CLI comes with Scrapy. Just run this command, and you are good to go:pip install scrapy```-Crawlee also [includes a CLI tool](https://crawlee.dev/docs/quick-start#installation-with-crawlee-cli) (`crawlee-cli`) that facilitates project setup, crawler creation and execution, streamlining the development process for users familiar with Node.js environments. The command for installation is:+Crawlee also [includes a CLI tool](https://crawlee.dev/js/docs/quick-start#installation-with-crawlee-cli) (`crawlee-cli`) that facilitates project setup, crawler creation and execution, streamlining the development process for users familiar with Node.js environments. The command for installation is:```bashnpx crawlee create my-crawler@@ -183,7 +183,7 @@ ROTATING_PROXY_LIST = [Now create a spider with the code you want to scrape any site and the `ROTATING_PROXY_LIST` in `settings.py` will manage which proxy to use for each request. Here middleware will treat each proxy initially as valid and then when a request is made, the middleware selects a proxy from the list of available proxies. The selection isn't purely sequential but is influenced by the recent history of proxy performance. The middleware has mechanisms to detect when a proxy might be banned or rendered ineffective. When such conditions are detected, the proxy is temporarily deactivated and put into a cooldown period. After the cooldown period expires, the proxy is reconsidered for use.-In Crawlee, you can [use your own proxy servers](https://crawlee.dev/docs/guides/proxy-management) or proxy servers acquired from third-party providers. If you already have your proxy URLs, you can start using them like this:+In Crawlee, you can [use your own proxy servers](https://crawlee.dev/js/docs/guides/proxy-management) or proxy servers acquired from third-party providers. If you already have your proxy URLs, you can start using them like this:```js title="crawler.js"import { ProxyConfiguration } from 'crawlee';@@ -200,7 +200,7 @@ const crawler = new CheerioCrawler({});```-Crawlee also has [`SessionPool`](https://crawlee.dev/api/core/class/SessionPool), a built-in allocation system for proxies. It handles the rotation, creation, and persistence of user-like sessions. It creates a pool of session instances that are randomly rotated.+Crawlee also has [`SessionPool`](https://crawlee.dev/js/api/core/class/SessionPool), a built-in allocation system for proxies. It handles the rotation, creation, and persistence of user-like sessions. It creates a pool of session instances that are randomly rotated.### Data storage@@ -263,7 +263,7 @@ Let's see how Crawlee stores the result:In Scrapy, handling anti-blocking strategies like [IP rotation](https://pypi.org/project/scrapy-rotated-proxy/), [user-agent rotation](https://python.plainenglish.io/rotating-user-agent-with-scrapy-78ca141969fe), custom solutions via middleware, and plugins are needed.-Crawlee provides HTTP crawling and [browser fingerprints](https://crawlee.dev/docs/guides/avoid-blocking) with zero configuration necessary; fingerprints are enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler` but also work with `CheerioCrawler` and the other HTTP Crawlers.+Crawlee provides HTTP crawling and [browser fingerprints](https://crawlee.dev/js/docs/guides/avoid-blocking) with zero configuration necessary; fingerprints are enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler` but also work with `CheerioCrawler` and the other HTTP Crawlers.### Error handling@@ -290,11 +290,11 @@ const crawler = new CheerioCrawler({There is also `noRetry`. If set to `true` then the request will not be automatically tried.-Crawlee also provides a built-in [logging mechanism](https://crawlee.dev/api/core/class/Log) via `log`, allowing you to log warnings, errors, and other information effectively.+Crawlee also provides a built-in [logging mechanism](https://crawlee.dev/js/api/core/class/Log) via `log`, allowing you to log warnings, errors, and other information effectively.### Deployment using Docker-Scrapy can be containerized using Docker, though it typically requires manual setup to create Dockerfiles and configure environments. While Crawlee includes [ready-to-use Docker configurations](https://crawlee.dev/docs/guides/docker-images), making deployment straightforward across various environments without additional configuration.+Scrapy can be containerized using Docker, though it typically requires manual setup to create Dockerfiles and configure environments. While Crawlee includes [ready-to-use Docker configurations](https://crawlee.dev/js/docs/guides/docker-images), making deployment straightforward across various environments without additional configuration.## Community@@ -312,4 +312,4 @@ But if you want to explore or are comfortable working with TypeScript or JavaScrAs promised, this is just the first of the many articles comparing Scrapy and Crawlee. With the upcoming articles, you will learn more about every technical detail.-Meanwhile, if you want to learn more about Crawlee, read our [introduction to Crawlee](https://crawlee.dev/docs/introduction) or Apify's [Crawlee web scraping tutorial](https://blog.apify.com/crawlee-web-scraping-tutorial/).+Meanwhile, if you want to learn more about Crawlee, read our [introduction to Crawlee](https://crawlee.dev/js/docs/introduction) or Apify's [Crawlee web scraping tutorial](https://blog.apify.com/crawlee-web-scraping-tutorial/).diff --git a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.mdindex 7e74f2f82..d3a6ef5e6 100644--- a/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md+++ b/website/blog/2024/06-10-creating-a-netflix-show-recommender-using-crawlee-and-react/index.md@@ -67,7 +67,7 @@ Run this command to install `crawlee`:npm install crawlee```-Crawlee utilizes Cheerio for HTML parsing and scraping of static websites. While faster and [less resource-intensive](https://crawlee.dev/docs/guides/scaling-crawlers), it can only scrape websites that do not require JavaScript rendering, making it unsuitable for SPAs (single page applications).+Crawlee utilizes Cheerio for HTML parsing and scraping of static websites. While faster and [less resource-intensive](https://crawlee.dev/js/docs/guides/scaling-crawlers), it can only scrape websites that do not require JavaScript rendering, making it unsuitable for SPAs (single page applications).In this tutorial we can extract data from the HTML structure, so we will go with Cheerio but for extracting data from SPAs or JavaScript-rendered websites, Crawlee also supports headless browser libraries like [Playwright](https://playwright.dev/) and [Puppeteer](https://pptr.dev/)@@ -119,7 +119,7 @@ This will give the `genres` and `shows` array having list of genres and shows stNow we have all the data that we want for our project and it’s time to store or save the scraped data. To store the data, Crawlee comes with a `pushData()` method.-The [pushData()](https://crawlee.dev/docs/introduction/saving-data) method creates a storage folder in the project directory and stores the scraped data in JSON format.+The [pushData()](https://crawlee.dev/js/docs/introduction/saving-data) method creates a storage folder in the project directory and stores the scraped data in JSON format.```jsawait pushData({@@ -290,4 +290,4 @@ Project link - [https://github.com/ayush2390/web-show-recommender](https://githuIn this project, we used Crawlee to scrape Netflix; similarly, Crawlee can be used to scrape single application pages (SPAs) and JavaScript-rendered websites. The best part is all of this can be done while coding in JavaScript/TypeScript and using a single library.-If you want to learn more about Crawlee, go through the [documentation](https://crawlee.dev/docs/quick-start) and this step-by-step [Crawlee web scraping tutorial](https://blog.apify.com/crawlee-web-scraping-tutorial/) from Apify.+If you want to learn more about Crawlee, go through the [documentation](https://crawlee.dev/js/docs/quick-start) and this step-by-step [Crawlee web scraping tutorial](https://blog.apify.com/crawlee-web-scraping-tutorial/) from Apify.diff --git a/website/blog/2024/06-24-proxy-management-in-crawlee/index.md b/website/blog/2024/06-24-proxy-management-in-crawlee/index.mdindex 05fe20611..113b036a3 100644--- a/website/blog/2024/06-24-proxy-management-in-crawlee/index.md+++ b/website/blog/2024/06-24-proxy-management-in-crawlee/index.md@@ -77,9 +77,9 @@ await crawler.run();## How tiered proxies use Session Pool under the hood-A session pool is a way to manage multiple [sessions](https://crawlee.dev/api/core/class/Session) on a website so you can distribute your requests across them, reducing the chances of being detected and blocked. You can imagine each session like a different human user with its own IP address.+A session pool is a way to manage multiple [sessions](https://crawlee.dev/js/api/core/class/Session) on a website so you can distribute your requests across them, reducing the chances of being detected and blocked. You can imagine each session like a different human user with its own IP address.-When you use tiered proxies, each proxy tier works with the [session pool](https://crawlee.dev/api/core/class/SessionPool) to enhance request distribution and manage errors effectively.+When you use tiered proxies, each proxy tier works with the [session pool](https://crawlee.dev/js/api/core/class/SessionPool) to enhance request distribution and manage errors effectively.diff --git a/website/blog/2024/09-30-jsdom-based-scraping/index.md b/website/blog/2024/09-30-jsdom-based-scraping/index.mdindex 7c6671bbc..f7dfe05c4 100644--- a/website/blog/2024/09-30-jsdom-based-scraping/index.md+++ b/website/blog/2024/09-30-jsdom-based-scraping/index.md@@ -28,7 +28,7 @@ You will see a list of hashtags with their live ranking, the number of posts theOur goal here is to extract the top 100 hashtags from the list with the given filters.-The two possible approaches are to use [`CheerioCrawler`](https://crawlee.dev/docs/guides/cheerio-crawler-guide), and the second one will be browser-based scraping. Cheerio gives results faster but does not work with JavaScript-rendered websites.+The two possible approaches are to use [`CheerioCrawler`](https://crawlee.dev/js/docs/guides/cheerio-crawler-guide), and the second one will be browser-based scraping. Cheerio gives results faster but does not work with JavaScript-rendered websites.Cheerio is not the best option here as the [Creative Center](https://ads.tiktok.com/business/creativecenter/inspiration/popular/hashtag/pad/en) is a web application, and the data source is API, so we can only get the hashtags initially present in the HTML structure but not each of the 100 as we require.@@ -108,7 +108,7 @@ In this function, the main goal is to call `https://ads.tiktok.com/business/crea:::note-Before going ahead, I want to mention that Crawlee natively supports JSDOM using the [JSDOM Crawler](https://crawlee.dev/api/jsdom-crawler). It gives a framework for the parallel crawling of web pages using plain HTTP requests and jsdom DOM implementation. It uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth.+Before going ahead, I want to mention that Crawlee natively supports JSDOM using the [JSDOM Crawler](https://crawlee.dev/js/api/jsdom-crawler). It gives a framework for the parallel crawling of web pages using plain HTTP requests and jsdom DOM implementation. It uses raw HTTP requests to download web pages, it is very fast and efficient on data bandwidth.:::diff --git a/website/blog/2024/12-13-scrape-google-maps-using-python/index.md b/website/blog/2024/12-13-scrape-google-maps-using-python/index.mdindex 610d8ad54..afd10c251 100644--- a/website/blog/2024/12-13-scrape-google-maps-using-python/index.md+++ b/website/blog/2024/12-13-scrape-google-maps-using-python/index.md@@ -69,11 +69,11 @@ pip install crawlee "crawlee[playwright]"playwright install```-*If you're new to **Crawlee**, check out its easy-to-follow documentation. It’s available for both [Node.js](https://www.crawlee.dev/docs/quick-start) and [Python](https://www.crawlee.dev/python/docs/quick-start).*+*If you're new to **Crawlee**, check out its easy-to-follow documentation. It’s available for both [Node.js](https://www.crawlee.dev/js/docs/quick-start) and [Python](https://www.crawlee.dev/python/docs/quick-start).*:::note-Before going ahead with the project, I'd like to ask you to star Crawlee for Python on [GitHub](https://github.com/apify/crawlee-python/), it helps us to spread the word to fellow scraper developers.+Before going ahead with the project, I'd like to ask you to star Crawlee for Python on [GitHub](https://github.com/apify/crawlee-python/), it helps us to spread the word to fellow scraper developers.:::@@ -397,7 +397,7 @@ async def _scrape_listings(self, context) -> None:breakif new_items > 0:await self._load_more_items(page)-+print(f"\nFinished processing! Total items: {len(self.processed_names)}")except Exception as e:print(f"Error in scraping: {str(e)}")diff --git a/website/blog/2025/03-05-superscraper/index.md b/website/blog/2025/03-05-superscraper/index.mdindex dfe075b7b..2c3e2baf6 100644--- a/website/blog/2025/03-05-superscraper/index.md+++ b/website/blog/2025/03-05-superscraper/index.md@@ -6,7 +6,7 @@ image: './img/superscraper.webp'authors: [SauravJ, RadoC]----[SuperScraper](https://github.com/apify/super-scraper) is an open-source [Actor](https://docs.apify.com/platform/actors) that combines features from various web scraping services, including [ScrapingBee](https://www.scrapingbee.com/), [ScrapingAnt](https://scrapingant.com/), and [ScraperAPI](https://www.scraperapi.com/).+[SuperScraper](https://github.com/apify/super-scraper) is an open-source [Actor](https://docs.apify.com/platform/actors) that combines features from various web scraping services, including [ScrapingBee](https://www.scrapingbee.com/), [ScrapingAnt](https://scrapingant.com/), and [ScraperAPI](https://www.scraperapi.com/).A key capability is its standby mode, which runs the Actor as a persistent API server. This removes the usual start-up times - a common pain point in many systems - and lets users make direct API calls to interact with the system immediately.@@ -18,7 +18,7 @@ This blog explains how SuperScraper works, highlights its implementation details### What is SuperScraper?-SuperScraper transforms a traditional scraper into an API server. Instead of running with static inputs and waiting for completion, it starts only once, stays active, and listens for incoming requests.+SuperScraper transforms a traditional scraper into an API server. Instead of running with static inputs and waiting for completion, it starts only once, stays active, and listens for incoming requests.### How to enable standby mode@@ -32,7 +32,7 @@ The project uses Node.js `http` module to create a server that listens on the de### Handling multiple crawlers-SuperScraper processes user requests using multiple instances of Crawlee’s [`PlaywrightCrawler`](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler). Since each `PlaywrightCrawler` instance can only handle one proxy configuration, a separate crawler is created for each unique proxy setting.+SuperScraper processes user requests using multiple instances of Crawlee’s [`PlaywrightCrawler`](https://crawlee.dev/js/api/playwright-crawler/class/PlaywrightCrawler). Since each `PlaywrightCrawler` instance can only handle one proxy configuration, a separate crawler is created for each unique proxy setting.For example, if the user sends one request for “normal” proxies and one request with residential US proxies, a separate crawler needs to be created for each proxy configuration. Hence, to solve this, we store the crawlers in a key-value map, where the key is a stringified proxy configuration.@@ -43,7 +43,7 @@ const crawlers = new Map<string, PlaywrightCrawler>();Here’s a part of the code that gets executed when a new request from the user arrives; if the crawler for this proxy configuration exists in the map, it will be used. Otherwise, a new crawler gets created. Then, we add the request to the crawler’s queue so it can be processed.```ts-const key = JSON.stringify(crawlerOptions);+const key = JSON.stringify(crawlerOptions);const crawler = crawlers.has(key) ? crawlers.get(key)! : await createAndStartCrawler(crawlerOptions);await crawler.addRequests([request]);@@ -108,7 +108,7 @@ export function addResponse(responseId: string, response: ServerResponse) {Here’s the updated logic for fetching/creating the corresponding crawler for a given proxy configuration, with a call to store the response object:```ts-const key = JSON.stringify(crawlerOptions);+const key = JSON.stringify(crawlerOptions);const crawler = crawlers.has(key) ? crawlers.get(key)! : await createAndStartCrawler(crawlerOptions);addResponse(request.uniqueKey!, res);diff --git a/website/docusaurus.config.js b/website/docusaurus.config.jsindex 96ca73d36..308af0881 100644--- a/website/docusaurus.config.js+++ b/website/docusaurus.config.js@@ -1,4 +1,4 @@-/* eslint-disable global-require,import/no-extraneous-dependencies */+/* eslint-disable global-require */const { externalLinkProcessor } = require('./tools/utils/externalLink');const packages = [@@ -34,7 +34,7 @@ const packagesOrder = [/** @type {Partial<import('@docusaurus/types').DocusaurusConfig>} */module.exports = {- title: 'Crawlee · Build reliable crawlers. Fast.',+ title: 'Crawlee for JavaScript · Build reliable crawlers. Fast.',tagline: 'Build reliable crawlers. Fast.',url: 'https://crawlee.dev',baseUrl: '/',@@ -66,6 +66,7 @@ module.exports = {showLastUpdateAuthor: true,showLastUpdateTime: true,path: '../docs',+ routeBasePath: 'js/docs',sidebarPath: './sidebars.js',rehypePlugins: [externalLinkProcessor],disableVersioning: !!process.env.CRAWLEE_DOCS_FAST,@@ -107,6 +108,7 @@ module.exports = {projectRoot: `${__dirname}/..`,changelogs: true,readmes: true,+ routeBasePath: 'js/api',sortPackages: (a, b) => {return packagesOrder.indexOf(a.packageName) - packagesOrder.indexOf(b.packageName);},@@ -121,33 +123,33 @@ module.exports = {{redirects: [{- from: '/docs',- to: '/docs/quick-start',+ from: '/js/docs',+ to: '/js/docs/quick-start',},{- from: '/docs/next',- to: '/docs/next/quick-start',+ from: '/js/docs/next',+ to: '/js/docs/next/quick-start',},{- from: '/docs/guides/environment-variables',- to: '/docs/guides/configuration',+ from: '/js/docs/guides/environment-variables',+ to: '/js/docs/guides/configuration',},{- from: '/docs/guides/getting-started',- to: '/docs/introduction',+ from: '/js/docs/guides/getting-started',+ to: '/js/docs/introduction',},{- from: '/docs/guides/apify-platform',- to: '/docs/deployment/apify-platform',+ from: '/js/docs/guides/apify-platform',+ to: '/js/docs/deployment/apify-platform',},],- createRedirects(existingPath) {- if (!existingPath.endsWith('/')) {- return `${existingPath}/`;- }-- return undefined; // Return a falsy value: no redirect created- },+ // createRedirects(existingPath) {+ // if (!existingPath.endsWith('/')) {+ // return `${existingPath}/`;+ // }+ //+ // return undefined; // Return a falsy value: no redirect created+ // },},],[@@ -170,6 +172,43 @@ module.exports = {},};},+ // skipping svgo for animated crawlee logo+ async function doNotUseSVGO() {+ return {+ name: 'docusaurus-svgo',+ configureWebpack(config) {+ // find the svg rule+ const svgRule = config.module.rules.find((r) => typeof r === 'object' && r.test.toString() === '/\\.svg$/i');++ // find the svgr loader+ const svgrLoader = svgRule?.oneOf?.[0];++ // make copy of svgr loader and disable svgo+ const svgrLoaderCopy = JSON.parse(JSON.stringify(svgrLoader));++ // include only animated logo+ svgrLoaderCopy.include = /animated-crawlee-logo/;++ // turn off svgo+ svgrLoaderCopy.use[0].options.svgo = false;++ // insert the copy after the original svgr loader+ svgRule.oneOf.splice(1, 0, svgrLoaderCopy);++ // exclude animated logo from the first svgr loader (with svgo enabled)+ svgrLoader.exclude = /animated-crawlee-logo/;++ return {+ mergeStrategy: {+ 'module.rules': 'replace',+ },+ module: {+ rules: config.module.rules,+ },+ };+ },+ };+ },],themeConfig:/** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({@@ -223,53 +262,6 @@ module.exports = {label: 'Blog',position: 'left',},- {- type: 'dropdown',- label: 'Node.js',- position: 'left',- items: [- {- label: 'Node.js',- href: '#',- target: '_self',- rel: 'dofollow',- },- {- label: 'Python',- href: 'https://crawlee.dev/python',- target: '_self',- rel: 'dofollow',- },- ],- },- {- type: 'docsVersionDropdown',- position: 'left',- dropdownItemsAfter: [- {- href: 'https://sdk.apify.com/docs/guides/getting-started',- label: '2.2',- },- {- href: 'https://sdk.apify.com/docs/1.3.1/guides/getting-started',- label: '1.3',- },- ],- },- {- href: 'https://github.com/apify/crawlee',- label: 'GitHub',- title: 'View on GitHub',- position: 'right',- className: 'icon',- },- {- href: 'https://discord.com/invite/jyEM2PRvMU',- label: 'Discord',- title: 'Chat on Discord',- position: 'right',- className: 'icon',- },],},colorMode: {@@ -297,29 +289,25 @@ module.exports = {items: [{label: 'Guides',- to: 'docs/guides',+ to: 'js/docs/guides',},{label: 'Examples',- to: 'docs/examples',+ to: 'js/docs/examples',},{label: 'API reference',- to: 'api/core',+ to: 'js/api/core',},{- label: 'Upgrading to v3',- to: 'docs/upgrading/upgrading-to-v3',+ label: 'Changelog',+ to: 'js/api/core/changelog',},],},{- title: 'Community',+ title: 'Product',items: [- {- label: 'Blog',- to: 'blog',- },{label: 'Discord',href: 'https://discord.com/invite/jyEM2PRvMU',@@ -332,13 +320,18 @@ module.exports = {label: 'Twitter',href: 'https://twitter.com/apify',},+ {+ label: 'YouTube',+ href: 'https://www.youtube.com/apify',+ },+],},{title: 'More',items: [{- label: 'Apify Platform',+ label: 'Apify platform',href: 'https://apify.com',},{@@ -352,20 +345,20 @@ module.exports = {],},],- logo: {- src: 'img/apify_logo.svg',- href: '/',- width: '60px',- height: '60px',- },},algolia: {appId: '5JC94MPMLY',apiKey: '267679200b833c2ca1255ab276731869', // search only (public) API keyindexName: 'crawlee',+ placeholder: 'Search documentation',algoliaOptions: {facetFilters: ['version:VERSION'],},+ translations: {+ button: {+ buttonText: 'Search documentation...',+ },+ },},}),};diff --git a/website/package.json b/website/package.jsonindex df585b4a2..798d68816 100644--- a/website/package.json+++ b/website/package.json@@ -53,6 +53,7 @@"raw-loader": "^4.0.2","react": "^19.0.0","react-dom": "^19.0.0",+ "react-github-btn": "^1.4.0","react-lite-youtube-embed": "^2.3.52","stream-browserify": "^3.0.0","unist-util-visit": "^5.0.0"diff --git a/website/src/components/ApiLink.jsx b/website/src/components/ApiLink.jsxindex 5b4a1f91f..947584c85 100644--- a/website/src/components/ApiLink.jsx+++ b/website/src/components/ApiLink.jsx@@ -15,12 +15,12 @@ const ApiLink = ({ to, children }) => {if (siteConfig.presets[0][1].docs.disableVersioning || version.version === stable) {return (- <Link to={`/api/${to}`}>{children}</Link>+ <Link to={`/js/api/${to}`}>{children}</Link>);}return (- <Link to={`/api/${version.version === 'current' ? 'next' : version.version}/${to}`}>{children}</Link>+ <Link to={`/js/api/${version.version === 'current' ? 'next' : version.version}/${to}`}>{children}</Link>);};diff --git a/website/src/components/Button.jsx b/website/src/components/Button.jsxnew file mode 100644index 000000000..453a0a81d--- /dev/null+++ b/website/src/components/Button.jsx@@ -0,0 +1,23 @@+import Link from '@docusaurus/Link';+import clsx from 'clsx';+import React from 'react';++import styles from './Button.module.css';+import CrawleeSvg from '../../static/img/crawlee-logo-monocolor.svg';++export default function Button({ children, to, withIcon, type = 'primary', className, isBig }) {+ return (+ <Link to={to} target="_self" rel="dofollow">+ <span className={clsx(+ className,+ styles.button,+ type === 'primary' && styles.buttonPrimary,+ type === 'secondary' && styles.buttonSecondary,+ isBig && styles.big,+ )}>+ {withIcon && <CrawleeSvg />}+ {children}+ </span>+ </Link>+ );+}diff --git a/website/src/components/Button.module.css b/website/src/components/Button.module.cssnew file mode 100644index 000000000..e1cd82da6--- /dev/null+++ b/website/src/components/Button.module.css@@ -0,0 +1,63 @@+.button {+ display: inline-flex;+ align-items: center;+ text-align: center;+ padding: 8px 16px;+ border-radius: 8px;+ font-family: (--ifm-font-family-base);+ font-size: 16px;+ font-style: normal;+ font-weight: 500;+ line-height: 24px;+ cursor: pointer;+ transition: background-color 0.2s;++ svg {+ margin-right: 8px;+ }+}++.buttonPrimary {+ background-color: var(--color-black-action);+ color: var(--color-text-on-primary);+ border: none;++ path {+ stroke: var(--color-text-on-primary);+ &:first-child {+ fill: var(--color-text-on-primary);+ }+ }+}++.buttonPrimary:hover {+ background-color: var(--color-primary-action-hover);+}++.buttonSecondary {+ background-color: var(--color-background);+ color: var(--color-text);+ border: 1px solid var(--color-border);++ path {+ stroke: var(--color-black-action);+ &:first-child {+ fill: var(--color-black-action);+ }+ }+}++.buttonSecondary:hover {+ border: 1px solid var(--color-text);+}++.big {+ padding: 12px 24px;+}++/* TABLET */+@media (min-width: 768px) {+ .button {+ width: auto;+ }+}diff --git a/website/src/components/CopyButton.jsx b/website/src/components/CopyButton.jsxnew file mode 100644index 000000000..073e66fbd--- /dev/null+++ b/website/src/components/CopyButton.jsx@@ -0,0 +1,34 @@+/* eslint-disable max-len */+import clsx from 'clsx';+import React, { useState } from 'react';++import styles from './CopyButton.module.css';++export default function CopyButton({ copyText, compact = false, className }) {+ const [copied, setCopied] = useState(false);+ const copy = async () => {+ await navigator.clipboard.writeText(copyText);+ setCopied(true);+ setTimeout(() => setCopied(false), 2000);+ };+ return <button+ type="button"+ aria-label="Copy code to clipboard"+ title="Copy"+ onClick={copy}+ className={clsx(className, styles.copyButton, compact ? styles.copyButtonCompact : styles.copyButtonDefault)}+ >+ {copied+ ? <svg width="20" height="20" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg">+ <path fillRule="evenodd" clipRule="evenodd" d="M18.0303 5.09467C18.3232 5.38756 18.3232 5.86244 18.0303 6.15533L8.03033 16.1553C7.73744 16.4482 7.26256 16.4482 6.96967 16.1553L2.59467 11.7803C2.30178 11.4874 2.30178 11.0126 2.59467 10.7197C2.88756 10.4268 3.36244 10.4268 3.65533 10.7197L7.5 14.5643L16.9697 5.09467C17.2626 4.80178 17.7374 4.80178 18.0303 5.09467Z" />+ </svg>++ : <svg width="20" height="20" viewBox="0 0 20 20" xmlns="http://www.w3.org/2000/svg">+ <path+ fillRule="evenodd"+ clipRule="evenodd"+ d="M8.375 2.375C7.13236 2.375 6.125 3.38236 6.125 4.625V6.125H4.625C3.38236 6.125 2.375 7.13236 2.375 8.375V15.375C2.375 16.6176 3.38236 17.625 4.625 17.625H11.625C12.8676 17.625 13.875 16.6176 13.875 15.375V13.875H15.375C16.6176 13.875 17.625 12.8676 17.625 11.625V4.625C17.625 3.38236 16.6176 2.375 15.375 2.375H8.375ZM13.875 12.375H15.375C15.7892 12.375 16.125 12.0392 16.125 11.625V4.625C16.125 4.21079 15.7892 3.875 15.375 3.875H8.375C7.96079 3.875 7.625 4.21079 7.625 4.625V6.125H11.625C12.8676 6.125 13.875 7.13236 13.875 8.375V12.375ZM4.625 7.625C4.21079 7.625 3.875 7.96079 3.875 8.375V15.375C3.875 15.7892 4.21079 16.125 4.625 16.125H11.625C12.0392 16.125 12.375 15.7892 12.375 15.375V8.375C12.375 7.96079 12.0392 7.625 11.625 7.625H4.625Z" />+ </svg>+ }+ </button>;+}diff --git a/website/src/components/CopyButton.module.css b/website/src/components/CopyButton.module.cssnew file mode 100644index 000000000..cb18577f4--- /dev/null+++ b/website/src/components/CopyButton.module.css@@ -0,0 +1,37 @@+.copyButton {+ all: unset;+ display: inline-flex;+ align-items: center;+ justify-content: center;+ box-sizing: border-box;+ cursor: pointer;+ fill: var(--color-icon);++ svg {+ flex-shrink: 0;+ }+}++.copyButtonDefault {+ width: 28px;+ height: 28px;+ background-color: var(--color-background-muted);+ border: 1px solid var(--color-border);+ border-radius: 6px;+ transition: background-color 0.12s ease-out;++ &:hover {+ background-color: var(--color-hover);+ }++ svg {+ padding: 1px;+ }+}++.copyButtonCompact {+ svg {+ width: 16px;+ height: 16px;+ }+}\ No newline at end of filediff --git a/website/src/components/Highlights.jsx b/website/src/components/Highlights.jsxindex 6c561f044..4f43fca63 100644--- a/website/src/components/Highlights.jsx+++ b/website/src/components/Highlights.jsx@@ -10,7 +10,7 @@ const FeatureList = [description: (<>We believe websites are best scraped in the language they're written in. Crawlee <b>runs on Node.js- and it's <a href="https://crawlee.dev/docs/guides/typescript-project">built in TypeScript</a></b> to improve code completion in your IDE,+ and it's <a href="https://crawlee.dev/js/docs/guides/typescript-project">built in TypeScript</a></b> to improve code completion in your IDE,even if you don't use TypeScript yourself. Crawlee supports both TypeScript and JavaScript crawling.</>),@@ -20,10 +20,10 @@ const FeatureList = [Svg: require('../../static/img/features/fingerprints.svg').default,description: (<>- Crawlee makes HTTP requests that <a href="https://crawlee.dev/docs/guides/avoid-blocking"><b>mimic browser headers and TLS fingerprints</b></a>.+ Crawlee makes HTTP requests that <a href="https://crawlee.dev/js/docs/guides/avoid-blocking"><b>mimic browser headers and TLS fingerprints</b></a>.It also rotates them automatically based on data about real-world traffic. Popular HTML- parsers <b><a href="https://crawlee.dev/docs/guides/cheerio-crawler-guide">Cheerio</a> - and <a href="https://crawlee.dev/docs/guides/jsdom-crawler-guide">JSDOM</a></b> are included.+ parsers <b><a href="https://crawlee.dev/js/docs/guides/cheerio-crawler-guide">Cheerio</a> + and <a href="https://crawlee.dev/js/docs/guides/jsdom-crawler-guide">JSDOM</a></b> are included.</>),},@@ -32,7 +32,7 @@ const FeatureList = [Svg: require('../../static/img/features/works-everywhere.svg').default,description: (<>- Switch your crawlers from HTTP to <a href="https://crawlee.dev/docs/guides/javascript-rendering">headless browsers</a> in 3 lines of code. Crawlee builds on top of <b>Puppeteer+ Switch your crawlers from HTTP to <a href="https://crawlee.dev/js/docs/guides/javascript-rendering">headless browsers</a> in 3 lines of code. Crawlee builds on top of <b>Puppeteerand Playwright</b> and adds its own <b>anti-blocking features and human-like fingerprints</b>. Chrome, Firefox and more.</>),@@ -42,8 +42,8 @@ const FeatureList = [Svg: require('../../static/img/features/auto-scaling.svg').default,description: (<>- Crawlee automatically manages concurrency based on <a href="https://crawlee.dev/api/core/class/AutoscaledPool">available system resources</a> and - <a href="https://crawlee.dev/docs/guides/proxy-management">smartly rotates proxies</a>.+ Crawlee automatically manages concurrency based on <a href="https://crawlee.dev/js/api/core/class/AutoscaledPool">available system resources</a> and + <a href="https://crawlee.dev/js/docs/guides/proxy-management">smartly rotates proxies</a>.Proxies that often time-out, return network errors or bad HTTP codes like 401 or 403 are discarded.</>),@@ -53,8 +53,8 @@ const FeatureList = [Svg: require('../../static/img/features/storage.svg').default,description: (<>- You can <a href="https://crawlee.dev/docs/guides/result-storage">save files, screenshots and JSON results</a> to disk with one line of code- or plug an adapter for your DB. Your URLs are <a href="https://crawlee.dev/docs/guides/request-storage">kept in a queue</a> that ensures their+ You can <a href="https://crawlee.dev/js/docs/guides/result-storage">save files, screenshots and JSON results</a> to disk with one line of code+ or plug an adapter for your DB. Your URLs are <a href="https://crawlee.dev/js/docs/guides/request-storage">kept in a queue</a> that ensures theiruniqueness and that you don't lose progress when something fails.</>),@@ -64,9 +64,9 @@ const FeatureList = [Svg: require('../../static/img/features/node-requests.svg').default,description: (<>- Crawlee includes tools for <a href="https://crawlee.dev/api/utils/namespace/social">extracting social handles</a> or phone numbers, infinite scrolling, blocking- unwanted assets <a href="https://crawlee.dev/api/utils">and many more</a>. It works great out of the box, but also provides - <a href="https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions">rich configuration options</a>.+ Crawlee includes tools for <a href="https://crawlee.dev/js/api/utils/namespace/social">extracting social handles</a> or phone numbers, infinite scrolling, blocking+ unwanted assets <a href="https://crawlee.dev/js/api/utils">and many more</a>. It works great out of the box, but also provides + <a href="https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions">rich configuration options</a>.</>),},diff --git a/website/src/components/Homepage/HomepageCliExample.jsx b/website/src/components/Homepage/HomepageCliExample.jsxnew file mode 100644index 000000000..20285ed3d--- /dev/null+++ b/website/src/components/Homepage/HomepageCliExample.jsx@@ -0,0 +1,24 @@+import React from 'react';++import CopyButton from '../CopyButton';+import styles from './HomepageCliExample.module.css';++export default function CliExample({ example }) {+ return (+ <section className={styles.cliExampleSection}>+ <div className={styles.cliExampleTitle}>+ Or start with a template from our CLI+ </div>+ <code className={styles.cliExampleCodeBlock}>+ <pre>+ <span className={styles.cliCommandPrefix}>$</span>+ {example}+ <CopyButton copyText={example} />+ </pre>+ </code>+ <div className={styles.cliExampleSubtitle}>+ Built with 🤍 by Apify. Forever free and open-source.+ </div>+ </section>+ );+}diff --git a/website/src/components/Homepage/HomepageCliExample.module.css b/website/src/components/Homepage/HomepageCliExample.module.cssnew file mode 100644index 000000000..1664b48ce--- /dev/null+++ b/website/src/components/Homepage/HomepageCliExample.module.css@@ -0,0 +1,68 @@+.cliExampleSection {+ display: flex;+ flex-direction: column;+ justify-content: center;+ align-items: center;+ text-align: center;+ padding: 16px;+}++.cliExampleTitle {+ color: var(--color-text-muted);+ font-size: 18px;+ font-style: normal;+ font-weight: 400;+ line-height: 28px;+ margin-bottom: 16px;+}++.cliExampleCodeBlock {+ width: fit-content;+ height: fit-content;+ padding: 0;+ border: 0;+ margin-bottom: 18px;+ width: 100%;++ pre {+ margin: 0;+ width: 100%;+ padding: 8px 16px;+ background-color: var(--color-background-muted);+ border: 1px solid var(--color-border);+ display: flex;+ align-items: center;+ gap: 16px;+ font-size: 14px;+ line-height: 20px;++ button {+ margin-left: auto;+ }+ }++ .cliCommandPrefix {+ color: var(--color-text-muted);+ user-select: none;+ }++ /* TABLET */+ @media (min-width: 768px) {+ max-width: 526px;+ }+}++.cliExampleSubtitle {+ color: var(--color-text-subtle);+ font-size: 16px;+ font-style: normal;+ font-weight: 400;+ line-height: 24px;+}++/* TABLET */+@media (min-width: 768px) {+ .cliExampleSection {+ padding: 64px 0;+ }+}diff --git a/website/src/components/Homepage/HomepageCtaSection.jsx b/website/src/components/Homepage/HomepageCtaSection.jsxnew file mode 100644index 000000000..1762a0f3b--- /dev/null+++ b/website/src/components/Homepage/HomepageCtaSection.jsx@@ -0,0 +1,71 @@+import { useColorMode } from '@docusaurus/theme-common';+import clsx from 'clsx';+import React from 'react';++import AnimatedLogoDark from './animated-crawlee-logo-dark.svg';+import AnimatedLogoLight from './animated-crawlee-logo-light.svg';+import styles from './HomepageCtaSection.module.css';+import homepageStyles from '../../pages/index.module.css';+import Button from '../Button';++export default function HomepageCtaSection({ showJs, showPython }) {+ const { colorMode } = useColorMode();+ return (+ <section className={styles.ctaSection}>+ <h2 className={styles.ctaTitle}>Get started now!</h2>+ <div className={styles.ctaDescription}>+ Crawlee won’t fix broken selectors for you (yet), but it makes+ building and maintaining reliable crawlers faster and easier—so+ you can focus on what matters most.+ </div>+ <div+ className={clsx(styles.ctaButtonContainer, {+ [styles.ctaButtonContainerFullWidth]: showJs && showPython,+ })}+ >+ {showJs && (+ <Button+ to={showPython ? '/js' : '/js/docs/quick-start'}+ withIcon+ type={showJs && showPython ? 'secondary' : 'primary'}+ isBig+ >+ {showPython ? 'Get started with JS' : 'Get started'}+ </Button>+ )}+ {showPython && (+ <Button+ to="https://crawlee.dev/python"+ withIcon+ type="secondary"+ isBig+ >+ {showJs ? 'Get started with Python' : 'Get started'}+ </Button>+ )}+ </div>++ <div+ className={homepageStyles.fadedOutSeparator}+ id={styles.ctaFadedOutSeparator}+ />+ <div+ className={homepageStyles.fadedOutSeparatorVertical}+ id={styles.fadedOutSeparatorVerticalLeft}+ />+ <div+ className={homepageStyles.fadedOutSeparatorVertical}+ id={styles.fadedOutSeparatorVerticalRight}+ />+ <div+ className={homepageStyles.dashedDecorativeCircle}+ id={styles.ctaDashedCircleRight}+ />+ {colorMode === 'dark' ? (+ <AnimatedLogoDark className={styles.ctaImage} />+ ) : (+ <AnimatedLogoLight className={styles.ctaImage} />+ )}+ </section>+ );+}diff --git a/website/src/components/Homepage/HomepageCtaSection.module.css b/website/src/components/Homepage/HomepageCtaSection.module.cssnew file mode 100644index 000000000..23c3aa0da--- /dev/null+++ b/website/src/components/Homepage/HomepageCtaSection.module.css@@ -0,0 +1,114 @@+.ctaSection {+ position: relative;+ display: flex;+ flex-direction: column;+ justify-content: center;+ align-items: center;+ text-align: center;+ padding: 16px;+ padding-bottom: 0;+ gap: 24px;+ overflow: clip;+}++.ctaTitle {+ color: var(--color-text);+ font-family: 'Lota Grotesque';+ font-size: 36px;+ font-style: normal;+ font-weight: 400;+ line-height: 46px;+ margin: 0;+}++.ctaDescription {+ color: var(--color-text-muted);+ font-size: 16px;+ font-style: normal;+ font-weight: 400;+ line-height: 28px;+ max-width: 780px;+}++.ctaButtonContainer {+ display: flex;+ flex-direction: column;+ justify-content: center;+ align-items: center;+ text-align: center;+ gap: 16px;+ width: 100%;+}++.ctaButtonContainerFullWidth {+ a,+ a span {+ text-align: center;+ width: 100%;+ justify-content: center;+ }+}++.ctaImage {+ z-index: -1;+ margin-top: -90px;+ margin-bottom: -30px;+ min-height: 400px;+}++#ctaFadedOutSeparator {+ position: absolute;+ top: 370px;+ width: 100%;+ z-index: -2;+}++#fadedOutSeparatorVerticalLeft {+ position: absolute;+ left: 190px;+ bottom: 0;+ height: 100%;+ z-index: -2;+}++#fadedOutSeparatorVerticalRight {+ position: absolute;+ right: 190px;+ bottom: 0;+ height: 100%;+ z-index: -2;+}++#ctaDashedCircleRight {+ position: absolute;+ right: -120px;+ top: 370px;+ z-index: -2;+}++/* TABLET */+@media (min-width: 768px) {+ .ctaSection {+ padding-top: 80px;+ }++ .ctaTitle {+ font-size: 48px;+ line-height: 56px;+ }++ .ctaButtonContainer {+ flex-direction: row;+ }++ .ctaDescription {+ font-size: 18px;+ }++ .ctaButtonContainerFullWidth {+ a,+ a span {+ width: fit-content;+ }+ }+}diff --git a/website/src/components/Homepage/HomepageHeroSection.jsx b/website/src/components/Homepage/HomepageHeroSection.jsxnew file mode 100644index 000000000..8cdde06f3--- /dev/null+++ b/website/src/components/Homepage/HomepageHeroSection.jsx@@ -0,0 +1,31 @@+import React from 'react';++import styles from './HomepageHeroSection.module.css';+import homepageStyles from '../../pages/index.module.css';++export default function HomepageHeroSection() {+ return (+ <section className={styles.hero}>+ <h1 className={styles.heroTitle}>+ Build reliable web scrapers. Fast.+ </h1>+ <div+ className={homepageStyles.dashedSeparator}+ id={styles.separatorHeroHeader}+ />+ <p className={styles.heroSubtitle}>+ Crawlee is a web scraping library for JavaScript and Python. It+ handles blocking, crawling, proxies, and browsers for you.+ </p>+ <div+ className={homepageStyles.dashedSeparator}+ id={styles.separatorHeroHeader2}+ >+ <div+ className={homepageStyles.dashedDecorativeCircle}+ id={styles.heroDecorativeCircle}+ />+ </div>+ </section>+ );+}diff --git a/website/src/components/Homepage/HomepageHeroSection.module.css b/website/src/components/Homepage/HomepageHeroSection.module.cssnew file mode 100644index 000000000..24a33e356--- /dev/null+++ b/website/src/components/Homepage/HomepageHeroSection.module.css@@ -0,0 +1,81 @@+.hero {+ display: flex;+ flex-direction: column;+ align-items: center;+ justify-content: center;+ padding: 32px 0;+ h1 {+ padding-inline: 12px;+ }+}++.heroTitle {+ color: var(--color-text);+ font-size: 52px;+ line-height: 60px;+ font-weight: 400;+ text-align: center;+ margin: 0 0 16px 0;+}++.heroSubtitle {+ color: var(--color-text-muted);+ font-size: 18px;+ line-height: 28px;+ font-weight: 400;+ text-align: center;+ margin: 0 16px;+ max-width: 792px;+}++#separatorHeroHeader {+ display: none;+}++#separatorHeroHeader2 {+ display: none;+}++#heroDecorativeCircle {+ width: 60px;+ height: 60px;+ right: -60px;+ top: 0px;+}++/* TABLET */+@media (min-width: 768px) {+ .hero {+ padding: 64px 0 0 0;+ h1 {+ padding-inline: 24px;+ }+ }+ .heroTitle {+ font-size: 54px;+ line-height: 64px;+ margin: 0 16px 24px 16px;+ }+ .heroSubtitle {+ margin: 0 16px 30px 16px;+ }+ #separatorHeroHeader {+ display: none;+ }+ #separatorHeroHeader2 {+ display: block;+ }+}++/* DESKTOP */+@media (min-width: 1024px) {+ .hero {+ padding: 120px 0 0 0;+ }+ .heroSubtitle {+ margin: 30px 16px;+ }+ #separatorHeroHeader {+ display: block;+ }+}diff --git a/website/src/components/Homepage/LanguageInfoWidget.jsx b/website/src/components/Homepage/LanguageInfoWidget.jsxnew file mode 100644index 000000000..8d7176116--- /dev/null+++ b/website/src/components/Homepage/LanguageInfoWidget.jsx@@ -0,0 +1,60 @@+import { useColorMode } from '@docusaurus/theme-common';+import ThemedImage from '@theme/ThemedImage';+import clsx from 'clsx';+import React from 'react';+import GitHubButton from 'react-github-btn';++import Button from '../Button';+import CopyButton from '../CopyButton';+import styles from './LanguageInfoWidget.module.css';++export default function LanguageInfoWidget({+ language,+ command,+ to,+ githubUrl,+}) {+ const { colorMode } = useColorMode();+ return (+ <div className={styles.languageGetStartedContainer}>+ {language === 'JavaScript' && (+ <ThemedImage+ sources={{+ light: '/img/crawlee-javascript-light.svg',+ dark: '/img/crawlee-javascript-dark.svg',+ }}+ alt="Crawlee JavaScript"+ />+ )}+ {language === 'Python' && (+ <ThemedImage+ sources={{+ light: '/img/crawlee-python-light.svg',+ dark: '/img/crawlee-python-dark.svg',+ }}+ alt="Crawlee Python"+ />+ )}+ <div className={clsx(styles.buttonContainer)}>+ <Button to={to}>+ {command ? 'Learn more' : 'Get started'}+ </Button>+ <GitHubButton+ href={githubUrl}+ data-color-scheme={colorMode}+ data-show-count="true"+ aria-label="Star crawlee on GitHub"+ data-size="large"+ style={{ minHeight: '28px' }}+ >+ Star+ </GitHubButton>+ </div>+ {command && (+ <code className={styles.commandContainer}>+ {command} <CopyButton copyText={command} compact />+ </code>+ )}+ </div>+ );+}diff --git a/website/src/components/Homepage/LanguageInfoWidget.module.css b/website/src/components/Homepage/LanguageInfoWidget.module.cssnew file mode 100644index 000000000..f027432ad--- /dev/null+++ b/website/src/components/Homepage/LanguageInfoWidget.module.css@@ -0,0 +1,73 @@+.languageGetStartedContainer {+ margin: 0;+ display: flex;+ flex-direction: column;+ align-items: center;+ padding-inline: 12px;+}++.languageGetStartedContainer img {+ height: 40px;+ margin-bottom: 16px;+}++.buttonContainer {+ display: flex;+ flex-direction: column;+ align-items: center;+ gap: 16px;+ & > span {+ line-height: 0;+ }+ a,+ a span {+ min-width: 190px;+ text-align: center;+ justify-content: center;+ }+}++.buttonContainer:has(+ code) {+ margin-bottom: 16px;+ gap: 12px;+}++.commandContainer {+ margin: 0;+ padding: 0;+ color: var(--color-text);+ font-size: 12px;+ font-style: normal;+ font-weight: 400;+ line-height: 16px;+ background-color: transparent;+ border: 0;+ display: flex;+ align-items: center;+}++.commandContainer button {+ opacity: 0;+ transition: opacity var(--ifm-transition-fast) ease-in;+}++.commandContainer:hover button,+.commandContainer button:hover {+ opacity: 1;+}++/* TABLET */+@media (min-width: 768px) {+ .languageGetStartedContainer {+ margin: 24px 0 40px 0;+ }+ .buttonContainer:has(+ code) {+ flex-direction: row;+ }+ .buttonContainer:has(+ code) {+ a,+ a span {+ min-width: 0;+ }+ }+}diff --git a/website/src/components/Homepage/LanguageSwitch.jsx b/website/src/components/Homepage/LanguageSwitch.jsxnew file mode 100644index 000000000..7f497018d--- /dev/null+++ b/website/src/components/Homepage/LanguageSwitch.jsx@@ -0,0 +1,50 @@+import React, { useCallback, useEffect, useRef, useState } from 'react';+import styles from './LanguageSwitch.module.css';+import clsx from 'clsx';++export default function LanguageSwitch({+ options = ['JavaScript', 'Python'],+ defaultOption = 'JavaScript',+ onChange,+}) {+ const [activeOption, setActiveOption] = useState(defaultOption)+ const [backgroundStyle, setBackgroundStyle] = useState({})+ const optionRefs = useRef < (HTMLButtonElement | null)[] > ([])++ const updateBackgroundStyle = useCallback(() => {+ const activeIndex = options.indexOf(activeOption)+ const activeElement = optionRefs.current[activeIndex]+ if (activeElement) {+ const { offsetLeft, offsetWidth } = activeElement+ setBackgroundStyle({+ transform: `translateX(${offsetLeft}px)`,+ width: `${offsetWidth}px`,+ })+ }+ }, [activeOption, options])++ useEffect(() => {+ updateBackgroundStyle()+ }, [updateBackgroundStyle])++ const handleOptionClick = (option) => {+ setActiveOption(option)+ onChange?.(option)+ }++ return (+ <div className={styles.languageSwitch}>+ {options.map((option, index) => (+ <button+ key={option}+ ref={(el) => (optionRefs.current[index] = el)}+ className={clsx(styles.switchOption, option === activeOption && styles.active)}+ onClick={() => handleOptionClick(option)}+ >+ {option}+ </button>+ ))}+ <div className={styles.switchBackground} style={backgroundStyle} />+ </div>+ )+}diff --git a/website/src/components/Homepage/LanguageSwitch.module.css b/website/src/components/Homepage/LanguageSwitch.module.cssnew file mode 100644index 000000000..66f706ad1--- /dev/null+++ b/website/src/components/Homepage/LanguageSwitch.module.css@@ -0,0 +1,41 @@+.languageSwitch {+ z-index: 1;+ display: inline-flex;+ position: relative;+ background-color: var(--color-background-subtle);+ border-radius: 6px;+ padding: 4px;+}++.switchOption {+ position: relative;+ z-index: 1;+ padding: 6px 16px;+ font-size: 14px;+ font-weight: 500;+ color: var(--color-text-muted);+ background: none;+ border: none;+ cursor: pointer;+ transition: color 0.3s ease;+}++.switchOption:hover {+ color: var(--color-text);+}++.switchOption.active {+ color: var(--color-text);+}++.switchBackground {+ position: absolute;+ top: 4px;+ bottom: 4px;+ left: 0;+ border-radius: 6px;+ background-color: var(--color-background);+ transition:+ transform 0.3s ease,+ width 0.3s ease;+}diff --git a/website/src/components/Homepage/RiverSection.jsx b/website/src/components/Homepage/RiverSection.jsxnew file mode 100644index 000000000..0e77a5ead--- /dev/null+++ b/website/src/components/Homepage/RiverSection.jsx@@ -0,0 +1,22 @@+import Link from '@docusaurus/Link';+import clsx from 'clsx';+import React from 'react';++import styles from './RiverSection.module.css';++export default function RiverSection({ title, description, content, reversed, to }) {+ return (+ <div className={styles.riverWrapper}>+ <div className={clsx(styles.riverContainer, { [styles.riverReversed]: reversed })}>+ <div className={clsx(styles.riverSection, styles.riverText)}>+ <h3 className={styles.riverTitle}>{title}</h3>+ <p className={styles.riverDescription}>{description}</p>+ <Link className={styles.riverButton} to={to}>+ Learn more+ </Link>+ </div>+ <div className={clsx(styles.riverSection, styles.riverContent)}>{content}</div>+ </div>+ </div>+ );+}diff --git a/website/src/components/Homepage/RiverSection.module.css b/website/src/components/Homepage/RiverSection.module.cssnew file mode 100644index 000000000..093778074--- /dev/null+++ b/website/src/components/Homepage/RiverSection.module.css@@ -0,0 +1,173 @@+/* Base styles */+.riverWrapper {+ width: 100%;+ border-top: 1px solid var(--color-separator);+ border-bottom: 1px solid var(--color-separator);+}++.riverContainer {+ max-width: 1200px;+ margin: 0 auto;+ display: flex;+ flex-direction: column;++ /* Tablet layout */+ @media (min-width: 768px) {+ flex-direction: row;++ &.riverReversed {+ flex-direction: row-reverse;+ }+ }+}++.riverSection {+ width: 100%;++ /* Tablet layout */+ @media (min-width: 768px) {+ min-width: 0;+ flex-basis: 50%;+ flex-grow: 0;+ }+}++.riverText {+ padding: 24px 16px;++ /* Tablet layout */+ @media (min-width: 768px) {+ padding: 40px 32px;+ }++ /* Desktop layout */+ @media (min-width: 1024px) {+ padding: 48px 80px;+ }+}++/* Text styles */+.riverTitle {+ flex: 1;+ margin-top: 0;+ margin-bottom: 12px;+ font-size: 32px;+ font-weight: 400;+ line-height: 40px;++ /* Desktop layout */+ @media (min-width: 1024px) {+ max-width: 440px;+ }+}++.riverDescription {+ margin-bottom: 24px;+ color: var(--color-text-muted);+ font-size: 16px;+ line-height: 24px;++ /* Desktop layout */+ @media (min-width: 1024px) {+ max-width: 440px;+ }+}++.riverButton {+ cursor: pointer;+ padding: 8px 12px;+ background-color: transparent;+ border: 1px solid var(--color-border);+ border-radius: 12px;+ display: flex;+ align-items: center;+ justify-content: center;+ font-size: 16px;+ line-height: 24px;+ transition: background-color 0.12s ease-out;+ width: fit-content;+ color: var(--color-text);++ &:hover {+ background-color: var(--color-hover);+ color: var(--color-text);+ }++ path {+ stroke: var(--color-icon);+ }+}++.riverButton::after {+ content: '→';+ margin-inline: 4px;+ transition: margin 0.3s ease;+}++.riverButton:hover {+ color: var(--color-text);+ &::after {+ margin: 0 0 0 8px;+ }+}++.riverContent {+ min-height: 180px;+ background-color: var(--color-background-muted);+ border-top: 1px solid var(--color-separator);+ display: flex;+ flex-direction: column;+ overflow: hidden;++ img {+ max-height: 284px;+ object-fit: cover;+ height: 100%;+ width: 100%;+ margin-block: auto;+ }++ :global(.code-block) {+ flex-grow: 1;+ margin-bottom: 0;+ border-radius: 0;+ box-shadow: none;++ :global(div[class*='codeBlockContent']) {+ height: 100%;++ pre {+ height: 100%;+ display: flex;+ align-items: center;+ background: var(--color-background-muted) !important;+ }+ code {+ height: auto;+ font-size: 14px;+ background: var(--color-background-muted);+ padding: 16px 8px 16px 4px;+ min-width: initial;+ span::before {+ margin-right: 16px;+ color: var(--color-text-subtle);+ opacity: 1;+ }+ }+ }+ }++ /* Tablet layout */+ @media (min-width: 768px) {+ border-top: none;+ border-left: 1px solid var(--color-separator);+ }++ .riverReversed & {+ /* Tablet layout */+ @media (min-width: 768px) {+ border-left: none;+ border-right: 1px solid var(--color-separator);+ }+ }+}diff --git a/website/src/components/Homepage/ThreeCardsWithIcon.jsx b/website/src/components/Homepage/ThreeCardsWithIcon.jsxnew file mode 100644index 000000000..3cb60af16--- /dev/null+++ b/website/src/components/Homepage/ThreeCardsWithIcon.jsx@@ -0,0 +1,52 @@+import Link from '@docusaurus/Link';+import clsx from 'clsx';+import React from 'react';++import styles from './ThreeCardsWithIcon.module.css';++export default function ThreeCardsWithIcon({ cards }) {+ return (+ <div className={styles.cardsWrapper}>+ {cards?.map((card, index) => {+ const content = (+ <>+ <div className={styles.cardIcon}>{card.icon}</div>+ <h3 className={styles.cardTitle}>{card.title}</h3>+ <p className={styles.cardDescription}>+ {card.description}+ </p>+ {card.actionLink && (+ <Link+ to={card.actionLink.href}+ className={styles.cardAction}+ >+ {card.actionLink.text}+ </Link>+ )}+ </>+ );++ if (card.to) {+ return (+ <Link+ className={clsx(+ styles.cardItem,+ styles.cardItemLink,+ )}+ to={card.to}+ key={index}+ >+ {content}+ </Link>+ );+ }++ return (+ <div className={styles.cardItem} key={index}>+ {content}+ </div>+ );+ })}+ </div>+ );+}diff --git a/website/src/components/Homepage/ThreeCardsWithIcon.module.css b/website/src/components/Homepage/ThreeCardsWithIcon.module.cssnew file mode 100644index 000000000..9b9a37ab0--- /dev/null+++ b/website/src/components/Homepage/ThreeCardsWithIcon.module.css@@ -0,0 +1,103 @@+.cardsWrapper {+ display: flex;+ flex-direction: column;+ border-block: 1px solid var(--color-separator);++ @media (min-width: 768px) {+ flex-direction: row;+ }+}++/* Card styles */+.cardItem {+ display: flex;+ flex: 1;+ flex-direction: column;+ padding: 40px 24px;+ background: var(--color-card-background);+ transition: background 0.1s ease;++ border-bottom: 1px solid var(--color-separator);+ &:last-child {+ border-bottom: 0;+ }++ @media (min-width: 768px) {+ border-bottom: 0;+ border-right: 1px solid var(--color-separator);+ &:last-child {+ border-right: 0;+ }+ }+}++a.cardItem:hover {+ background: var(--color-card-background-hover);+}++.cardItem:has(:local(.cardAction)) {+ padding: 24px;+}++.cardIcon {+ margin-bottom: 16px;+ display: flex;+ align-items: center;+ justify-content: center;++ width: 72px;+ height: 72px;++ border-radius: 6px;+ border: 1px solid var(--color-separator);+ background: var(--color-background);+}++.cardIcon img {+ width: 50px;+}++.cardTitle {+ margin: 0;+ margin-bottom: 8px;+ color: var(--color-text);+ font-size: 26px;+ font-style: normal;+ font-weight: 400;+ line-height: 34px;+}++.cardDescription {+ color: var(--color-text-muted);+ font-family: var(--ifm-font-family-base);+ font-size: 16px;+ font-style: normal;+ font-weight: 400;+ line-height: 24px;+ margin: 0;+ margin-bottom: 12px;+}++.cardAction {+ color: var(--color-text-muted);+ font-family: var(--ifm-font-family-base);+ font-size: 16px;+ font-style: normal;+ font-weight: 650;+ line-height: 24px;+ width: fit-content;+ margin-top: auto;+}++.cardAction::after {+ content: '→';+ margin-left: 4px;+ transition: margin 0.3s ease;+}++.cardAction:hover {+ color: var(--color-text);+ &::after {+ margin-left: 8px;+ }+}diff --git a/website/src/components/Homepage/animated-crawlee-logo-dark.svg b/website/src/components/Homepage/animated-crawlee-logo-dark.svgnew file mode 100644index 000000000..f09395e3d--- /dev/null+++ b/website/src/components/Homepage/animated-crawlee-logo-dark.svg@@ -0,0 +1,275 @@+<svg width="543" height="615" viewBox="0 0 543 615" fill="none" xmlns="http://www.w3.org/2000/svg" >+<style>+ g > rect {+ transform: translate3d(0, 0, 0);+ animation: rotate 5s ease-in-out infinite;+ transform-origin: center;+ transform-box: fill-box;+ will-change: transform;+ }++ @keyframes rotate {+ 0% { transform: rotateZ(0deg); }+ 50% { transform: rotateZ(180deg); }+ 100% { transform: rotateZ(0deg); }+ }+</style>+<g filter="url(#filter14_dddd_964_28846)">+<mask id="mask0_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="238.25" y="444.375" width="59" height="40.5">+ <rect x="238.25" y="444.375" width="57" height="40.5" fill="url(#paint0_linear_964_28846)"/>+</mask>+<g mask="url(#mask0_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 266.579 463.493)">+<rect width="5.69774" height="5.69774" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 5.59" style="animation-delay: 0s;"/>+</g>+</g>+<mask id="mask1_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="197.25" y="401.875" width="129" height="85.5">+ <rect x="197.25" y="401.875" width="129" height="85.5" fill="url(#paint1_linear_964_28846)"/>+</mask>+<g mask="url(#mask1_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 253.686 441.837)">+<rect width="19.6416" height="19.6416" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 9.71" style="animation-delay: 0.05s;"/>+</g>+</g>+<mask id="mask2_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="212.5" y="387.5" width="119.5" height="74.25">+<rect x="212.5" y="387.5" width="118.5" height="74.25" fill="url(#paint2_linear_964_28846)"/>+</mask>+<g mask="url(#mask2_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 235.966 420.19)">+<rect width="38.8063" height="38.8063" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 12.83" style="animation-delay: 0.1s;"/>+</g>+</g>+<mask id="mask3_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="182.5" y="349.5" width="177" height="110.25">+<rect x="182.5" y="349.5" width="176" height="110.25" fill="url(#paint3_linear_964_28846)"/>+</mask>+<g mask="url(#mask3_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 218.268 398.543)">+<rect class="rotateRect" width="57.9474" height="57.9474" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 14.3868" style="animation-delay: 0.15s;"/>+</g>+</g>+<mask id="mask4_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="168.5" y="313.5" width="231" height="141.75">+<rect x="168.5" y="313.5" width="230" height="141.75" fill="url(#paint4_linear_964_28846)"/>+</mask>+<g mask="url(#mask4_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 200.251 375.993)">+<rect width="77.4321" height="77.4321" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.386" style="animation-delay: 0.2s;"/>+</g>+</g>+<mask id="mask5_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="152.5" y="277.5" width="285" height="173.25">+<rect x="152.5" y="277.5" width="284" height="173.25" fill="url(#paint5_linear_964_28846)"/>+</mask>+<g mask="url(#mask5_964_28846)" >+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 182.553 354.784)">+<rect width="96.5732" height="96.5732" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.995" style="animation-delay: 0.25s;"/>+</g>+</g>+<mask id="mask6_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="139.5" y="241.5" width="342" height="207">+<rect x="139.5" y="241.5" width="341" height="207" fill="url(#paint6_linear_964_28846)"/>+</mask>+<g mask="url(#mask6_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 165.445 332.699)">+<rect width="115.076" height="115.076" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.3394" style="animation-delay: 0.3s;"/>+</g>+</g>+<mask id="mask7_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="128.5" y="207" width="387" height="247.5">+<rect x="128.5" y="207" width="386" height="247.5" fill="url(#paint7_linear_964_28846)"/>+</mask>+<g mask="url(#mask7_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 147.5 307.437)">+<rect width="134.315" height="134.315" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.689" style="animation-delay: 0.35s;"/>+</g>+</g>+<mask id="mask8_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="139.5" y="194.5" width="342" height="225">+<rect x="139.5" y="194.5" width="341" height="225" fill="url(#paint8_linear_964_28846)" />+</mask>+<g mask="url(#mask8_964_28846)" >+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 165.666 285.719)">+<rect width="115.076" height="115.076" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.3394" style="animation-delay: 0.4s;"/>+</g>+</g>+<mask id="mask9_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="152.5" y="186.5" width="285" height="173.25">+<rect x="152.5" y="186.5" width="284" height="173.25" fill="url(#paint9_linear_964_28846)"/>+</mask>+<g mask="url(#mask9_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 182.773 263.566)">+<rect width="96.5732" height="96.5732" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.995" style="animation-delay: 0.45s;"/>+</g>+</g>+<mask id="mask10_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="168.5" y="179.5" width="231" height="141.75">+<rect x="168.5" y="179.5" width="230" height="141.75" fill="url(#paint10_linear_964_28846)"/>+</mask>+<g mask="url(#mask10_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 200.472 242.288)">+<rect width="77.4321" height="77.4321" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.386" style="animation-delay: 0.5s;"/>+</g>+</g>+<mask id="mask11_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="182.5" y="170.5" width="177" height="110.25">+<rect x="182.5" y="170.5" width="176" height="110.25" fill="url(#paint11_linear_964_28846)"/>+</mask>+<g mask="url(#mask11_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 218.488 219.668)">+<rect width="57.9474" height="57.9474" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 14.3868" style="animation-delay: 0.55s;"/>+</g>+</g>+<mask id="mask12_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="212.5" y="164.5" width="119.5" height="74.25">+<rect x="212.5" y="164.5" width="118.5" height="74.25" fill="url(#paint12_linear_964_28846)"/>+</mask>+<g mask="url(#mask12_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 236.187 197.951)">+<rect width="38.8063" height="38.8063" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 12.83" style="animation-delay: 0.6s;"/>+</g>+</g>+<mask id="mask13_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="197.25" y="136.875" width="129" height="85.5">+ <rect x="197.25" y="136.875" width="129" height="85.5" fill="url(#paint13_linear_964_28846)"/>+</mask>+<g mask="url(#mask13_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 253.907 176.234)">+<rect width="19.6416" height="19.6416" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 9.71" style="animation-delay: 0.65s;"/>+</g>+</g>+<mask id="mask14_964_28846" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="242" y="136.5" width="60" height="40.5">+ <rect x="242" y="136.5" width="60" height="40.5" fill="url(#paint14_linear_964_28846)"/>+</mask>+<g mask="url(#mask14_964_28846)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 266.8 154.527)">+<rect width="5.69774" height="5.69774" stroke="white" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 5.59" style="animation-delay: 0.7s;"/>+</g>+</g>+</g>+<g style="mix-blend-mode:screen" filter="url(#filter15_f_964_28846)">+<mask id="path-31-inside-1_964_28846" fill="white">+<path d="M271.5 150L392.886 307.07L271.5 464.139L150.114 307.07L271.5 150Z"/>+</mask>+<path d="M271.5 150L392.886 307.07L271.5 464.139L150.114 307.07L271.5 150Z" stroke="url(#paint15_linear_964_28846)" stroke-width="30" mask="url(#path-31-inside-1_964_28846)"/>+</g>+<g style="mix-blend-mode:screen" filter="url(#filter16_f_964_28846)">+<mask id="path-32-inside-2_964_28846" fill="white">+<path d="M271.5 150L392.886 307.07L271.5 464.139L150.114 307.07L271.5 150Z"/>+</mask>+<path d="M271.5 150L392.886 307.07L271.5 464.139L150.114 307.07L271.5 150Z" stroke="url(#paint16_linear_964_28846)" stroke-width="34" mask="url(#path-32-inside-2_964_28846)"/>+</g>+<defs>++<filter id="filter14_dddd_964_28846" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">+<feFlood flood-opacity="0" result="BackgroundImageFix"/>+<feColorMatrix in="SourceAlpha" type="matrix" result="hardAlpha"/>+<feGaussianBlur stdDeviation="1"/>+<feColorMatrix type="matrix" values="0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0"/>+<feBlend mode="normal" in2="BackgroundImageFix" result="effect1_dropShadow_964_28846"/>+<feColorMatrix type="matrix" values="0 0 0 0 0.895833 0 0 0 0 0.354167 0 0 0 0 1 0 0 0 1 0"/>+<feBlend mode="normal" in2="effect1_dropShadow_964_28846" result="effect2_dropShadow_964_28846"/>+<feGaussianBlur stdDeviation="12"/>+<feBlend mode="normal" in2="effect2_dropShadow_964_28846" result="effect3_dropShadow_964_28846"/>+<feGaussianBlur stdDeviation="20"/>+<feBlend mode="normal" in2="effect3_dropShadow_964_28846" result="effect4_dropShadow_964_28846"/>+<feBlend mode="normal" in="SourceGraphic" in2="effect4_dropShadow_964_28846" result="shape"/>+</filter>++<filter id="filter15_f_964_28846" x="90.1138" y="90" width="362.772" height="434.139" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">+<feFlood flood-opacity="0" result="BackgroundImageFix"/>+<feBlend mode="normal" in="SourceGraphic" in2="BackgroundImageFix" result="shape"/>+<feGaussianBlur stdDeviation="30" result="effect1_foregroundBlur_964_28846"/>+</filter>+<filter id="filter16_f_964_28846" x="0.11377" y="0" width="542.772" height="614.139" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">+<feFlood flood-opacity="0" result="BackgroundImageFix"/>+<feBlend mode="normal" in="SourceGraphic" in2="BackgroundImageFix" result="shape"/>+<feGaussianBlur stdDeviation="75" result="effect1_foregroundBlur_964_28846"/>+</filter>+<linearGradient id="paint0_linear_964_28846" x1="272" y1="459" x2="272" y2="468" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint1_linear_964_28846" x1="272" y1="432" x2="272" y2="451" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint2_linear_964_28846" x1="272" y1="404" x2="272" y2="437" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint3_linear_964_28846" x1="272" y1="374" x2="272" y2="423" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint4_linear_964_28846" x1="272" y1="345" x2="272" y2="408" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint5_linear_964_28846" x1="272" y1="316" x2="272" y2="393" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint6_linear_964_28846" x1="272" y1="287" x2="272" y2="379" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint7_linear_964_28846" x1="272" y1="252" x2="272" y2="362" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint8_linear_964_28846" x1="272" y1="240" x2="272" y2="332" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint9_linear_964_28846" x1="272" y1="225" x2="272" y2="302" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint10_linear_964_28846" x1="272" y1="211" x2="272" y2="274" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint11_linear_964_28846" x1="272" y1="195" x2="272" y2="244" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint12_linear_964_28846" x1="272" y1="181" x2="272" y2="214" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint13_linear_964_28846" x1="272" y1="167" x2="272" y2="186" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint14_linear_964_28846" x1="272" y1="150" x2="272" y2="159" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint15_linear_964_28846" x1="350.035" y1="385.604" x2="192.965" y2="228.535" gradientUnits="userSpaceOnUse">+<stop offset="0.001" stop-color="#FFB200"/>+<stop offset="0.0815" stop-color="#FFB100"/>+<stop offset="0.156" stop-color="#FFAF02"/>+<stop offset="0.2255" stop-color="#FEAB04"/>+<stop offset="0.2908" stop-color="#FDA606"/>+<stop offset="0.3528" stop-color="#FCA00A"/>+<stop offset="0.4126" stop-color="#FB980E"/>+<stop offset="0.471" stop-color="#FA9013"/>+<stop offset="0.5289" stop-color="#F98618"/>+<stop offset="0.5873" stop-color="#F77B1E"/>+<stop offset="0.6471" stop-color="#F56F24"/>+<stop offset="0.7091" stop-color="#F3632B"/>+<stop offset="0.7744" stop-color="#F25532"/>+<stop offset="0.8439" stop-color="#EF473A"/>+<stop offset="0.9184" stop-color="#ED3842"/>+<stop offset="0.999" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint16_linear_964_28846" x1="350.035" y1="385.604" x2="192.965" y2="228.535" gradientUnits="userSpaceOnUse">+<stop offset="0.001" stop-color="#FFB200"/>+<stop offset="0.0815" stop-color="#FFB100"/>+<stop offset="0.156" stop-color="#FFAF02"/>+<stop offset="0.2255" stop-color="#FEAB04"/>+<stop offset="0.2908" stop-color="#FDA606"/>+<stop offset="0.3528" stop-color="#FCA00A"/>+<stop offset="0.4126" stop-color="#FB980E"/>+<stop offset="0.471" stop-color="#FA9013"/>+<stop offset="0.5289" stop-color="#F98618"/>+<stop offset="0.5873" stop-color="#F77B1E"/>+<stop offset="0.6471" stop-color="#F56F24"/>+<stop offset="0.7091" stop-color="#F3632B"/>+<stop offset="0.7744" stop-color="#F25532"/>+<stop offset="0.8439" stop-color="#EF473A"/>+<stop offset="0.9184" stop-color="#ED3842"/>+<stop offset="0.999" stop-color="#EB284B"/>+</linearGradient>+</defs>+</svg>diff --git a/website/src/components/Homepage/animated-crawlee-logo-light.svg b/website/src/components/Homepage/animated-crawlee-logo-light.svgnew file mode 100644index 000000000..682a8d38e--- /dev/null+++ b/website/src/components/Homepage/animated-crawlee-logo-light.svg@@ -0,0 +1,222 @@+<svg width="543" height="615" viewBox="0 0 266 330" fill="none" xmlns="http://www.w3.org/2000/svg">+<style>+ g > rect {+ transform: translate3d(0, 0, 0);+ animation: rotate 5s ease-in-out infinite;+ transform-origin: center;+ transform-box: fill-box;+ will-change: transform;+ }++ @keyframes rotate {+ 0% { transform: rotateZ(0deg); }+ 50% { transform: rotateZ(180deg); }+ 100% { transform: rotateZ(0deg); }+ }+</style>+<g transform="scale(0.5347593583)translate(116 146)" filter="url(#filter8_dddd_962_28212)">+<mask id="mask0_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="98.5" y="310.5" width="140" height="45">+ <rect x="98.5" y="310.5" width="140" height="45" fill="url(#paint0_linear_962_28212)"/>+</mask>+<g mask="url(#mask0_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 127.579 317.493)">+<rect width="5.69774" height="5.69774" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 5.59" style="animation-delay: 0s;"/>+</g>+</g>+<mask id="mask1_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="89.5" y="276.5" width="220" height="95">+ <rect x="89.5" y="276.5" width="215" height="95" fill="url(#paint1_linear_962_28212)"/>+</mask>+<g mask="url(#mask1_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 114.686 295.837)">+<rect width="19.6416" height="19.6416" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 9.71" style="animation-delay: 0.05s;"/>+</g>+</g>+<mask id="mask2_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-187" y="123.5" width="400" height="200">+ <rect x="-186.5" y="123.5" width="395" height="200" fill="url(#paint2_linear_962_28212)"/>+</mask>+<g mask="url(#mask2_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 96.9658 274.19)">+<rect width="38.8063" height="38.8063" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 12.83" style="animation-delay: 0.1s;"/>+</g>+</g>+<mask id="mask3_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-169" y="103.5" width="590" height="245">+ <rect x="-168.5" y="103.5" width="585" height="245" fill="url(#paint3_linear_962_28212)"/>+</mask>+<g mask="url(#mask3_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 79.2673 252.543)">+<rect width="57.9474" height="57.9474" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 14.3868" style="animation-delay: 0.15s;"/>+</g>+</g>+<mask id="mask4_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-151" y="79" width="770" height="315">+ <rect x="-150.5" y="79" width="765" height="315" fill="url(#paint4_linear_962_28212)"/>+</mask>+<g mask="url(#mask4_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 61.2512 229.993)">+<rect width="77.4321" height="77.4321" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.386" style="animation-delay: 0.2s;"/>+</g>+</g>+<mask id="mask5_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-133" y="54.5" width="950" height="385">+ <rect x="-132.5" y="54.5" width="945" height="385" fill="url(#paint5_linear_962_28212)"/>+</mask>+<g mask="url(#mask5_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 43.553 208.784)">+<rect width="96.5732" height="96.5732" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.995" style="animation-delay: 0.25s;"/>+</g>+</g>+<mask id="mask6_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-114" y="29.5" width="1140" height="460">+ <rect x="-113.5" y="29.5" width="1135" height="460" fill="url(#paint6_linear_962_28212)"/>+</mask>+<g mask="url(#mask6_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 26.4451 186.699)">+<rect width="115.076" height="115.076" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.3394" style="animation-delay: 0.3s;"/>+</g>+</g>+<mask id="mask7_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-99" y="-5" width="1290" height="550">+ <rect x="-98.5" y="-5" width="1285" height="550" fill="url(#paint7_linear_962_28212)"/>+</mask>+<g mask="url(#mask7_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 8.5 161.437)">+<rect width="134.315" height="134.315" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.689" style="animation-delay: 0.35s;"/>+</g>+</g>+<mask id="mask8_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-114" y="-11" width="1140" height="460">+ <rect x="-113.5" y="-11" width="1135" height="460" fill="url(#paint8_linear_962_28212)"/>+</mask>+<g mask="url(#mask8_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 26.666 139.719)">+<rect width="115.076" height="115.076" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 16.3394" style="animation-delay: 0.4s;"/>+</g>+</g>+<mask id="mask9_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-133" y="-28.5" width="950" height="385">+ <rect x="-132.5" y="-28.5" width="945" height="385" fill="url(#paint9_linear_962_28212)"/>+</mask>+<g mask="url(#mask9_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 43.7737 117.566)">+<rect width="96.5732" height="96.5732" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.995" style="animation-delay: 0.45s;"/>+</g>+</g>+<mask id="mask10_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-151" y="-44" width="770" height="315">+ <rect x="-150.5" y="-44" width="765" height="315" fill="url(#paint10_linear_962_28212)"/>+</mask>+<g mask="url(#mask10_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 61.4722 96.2881)">+<rect width="77.4321" height="77.4321" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 15.386" style="animation-delay: 0.5s;"/>+</g>+</g>+<mask id="mask11_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-169" y="-72.5" width="590" height="245">+ <rect x="-168.5" y="-72.5" width="585" height="245" fill="url(#paint11_linear_962_28212)"/>+</mask>+<g mask="url(#mask11_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 79.4883 73.668)">+<rect width="57.9474" height="57.9474" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 14.3868" style="animation-delay: 0.55s;"/>+</g>+</g>+<mask id="mask12_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="-187" y="-107.5" width="400" height="200">+ <rect x="-186.5" y="-107.5" width="395" height="200" fill="url(#paint12_linear_962_28212)"/>+</mask>+<g mask="url(#mask12_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 97.1863 51.9512)">+<rect width="38.8063" height="38.8063" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 12.83" style="animation-delay: 0.6s;"/>+</g>+</g>+<mask id="mask13_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="89.5" y="-1.5" width="220" height="95">+ <rect x="89.5" y="-1.5" width="215" height="95" fill="url(#paint13_linear_962_28212)"/>+</mask>+<g mask="url(#mask13_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 114.907 30.2344)">+<rect width="19.6416" height="19.6416" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 9.71" style="animation-delay: 0.65s;"/>+</g>+</g>+<mask id="mask14_962_28212" style="mask-type:alpha" maskUnits="userSpaceOnUse" x="98.5" y="-8.5" width="140" height="45">+ <rect x="98.5" y="-8.5" width="140" height="45" fill="url(#paint14_linear_962_28212)"/>+</mask>+<g mask="url(#mask14_962_28212)">+<g transform="matrix(0.92388 -0.382683 0.925375 0.379052 127.8 8.52734)">+<rect width="5.69774" height="5.69774" stroke="#F25532" stroke-width="3" stroke-linecap="round" stroke-dasharray="0.1 5.59" style="animation-delay: 0.7s;"/>+</g>+</g>+</g>+<defs>+<filter id="filter8_dddd_962_28212" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">+<feFlood flood-opacity="0" result="BackgroundImageFix"/>+<feColorMatrix in="SourceAlpha" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0" result="hardAlpha"/>+<feOffset/>+<feGaussianBlur stdDeviation="1"/>+<feColorMatrix type="matrix" values="0 0 0 0 0.947194 0 0 0 0 0.333676 0 0 0 0 0.198024 0 0 0 1 0"/>+<feBlend mode="normal" in2="BackgroundImageFix" result="effect1_dropShadow_962_28212"/>+<feOffset/>+<feGaussianBlur stdDeviation="3.5"/>+<feColorMatrix type="matrix" values="0 0 0 0 0.895833 0 0 0 0 0.354167 0 0 0 0 1 0 0 0 1 0"/>+<feBlend mode="normal" in2="effect1_dropShadow_962_28212" result="effect2_dropShadow_962_28212"/>+<feOffset/>+<feGaussianBlur stdDeviation="12"/>+<feBlend mode="normal" in2="effect2_dropShadow_962_28212" result="effect3_dropShadow_962_28212"/>+<feColorMatrix in="SourceAlpha" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32 0" result="hardAlpha"/>+<feOffset/>+<feGaussianBlur stdDeviation="10"/>+<feColorMatrix type="matrix" values="0 0 0 0 0.894118 0 0 0 0 0.352941 0 0 0 0 1 0 0 0 1 0"/>+<feBlend mode="normal" in2="effect3_dropShadow_962_28212" result="effect4_dropShadow_962_28212"/>+<feBlend mode="normal" in="SourceGraphic" in2="effect4_dropShadow_962_28212" result="shape"/>+</filter>+<linearGradient id="paint0_linear_962_28212" x1="133" y1="313" x2="133" y2="322" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint1_linear_962_28212" x1="133" y1="286" x2="133" y2="305" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint2_linear_962_28212" x1="133" y1="258" x2="133" y2="291" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint3_linear_962_28212" x1="133" y1="228" x2="133" y2="277" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint4_linear_962_28212" x1="133" y1="199" x2="133" y2="262" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint5_linear_962_28212" x1="133" y1="170" x2="133" y2="247" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint6_linear_962_28212" x1="133" y1="141" x2="133" y2="233" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint7_linear_962_28212" x1="133" y1="106" x2="133" y2="216" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint8_linear_962_28212" x1="133" y1="94" x2="133" y2="186" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint9_linear_962_28212" x1="133" y1="79" x2="133" y2="156" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint10_linear_962_28212" x1="133" y1="65" x2="133" y2="128" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint11_linear_962_28212" x1="133" y1="49" x2="133" y2="98" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint12_linear_962_28212" x1="133" y1="35" x2="133" y2="68" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint13_linear_962_28212" x1="133" y1="21" x2="133" y2="40" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+<linearGradient id="paint14_linear_962_28212" x1="133" y1="4" x2="133" y2="13" gradientUnits="userSpaceOnUse">+<stop stop-color="#D9D9D9" stop-opacity="0"/>+<stop offset="1" stop-color="#D9D9D9"/>+</linearGradient>+</defs>+</svg>diff --git a/website/src/components/RunnableCodeBlock.module.css b/website/src/components/RunnableCodeBlock.module.cssindex 5cbeabc56..fce26f125 100644--- a/website/src/components/RunnableCodeBlock.module.css+++ b/website/src/components/RunnableCodeBlock.module.css@@ -2,7 +2,7 @@display: inline-block;padding: 3px 10px;position: absolute;- top: 9px;+ top: calc(var(--ifm-pre-padding) / 2);right: 9px;z-index: 1;font-size: 16px;diff --git a/website/src/css/custom.css b/website/src/css/custom.cssindex 5c725e471..336d084ee 100644--- a/website/src/css/custom.css+++ b/website/src/css/custom.css@@ -2,7 +2,7 @@html[data-theme='dark'] {--ifm-navbar-background-color: #1a1b23;- --ifm-background-color: #1a1b23;+ --ifm-background-color: #1a1b21;--ifm-background-surface-color: #242736;--ifm-font-color-base: #f2f3fb;@@ -13,24 +13,51 @@ html[data-theme='dark'] {--ifm-link-color: #5d9df1;--ifm-heading-color: #f2f3fb;--ifm-navbar-link-color: #f2f3fb;-- --docsearch-text-color: #8d92af;+ --ifm-menu-color-active: #b2b8cc;++ --docsearch-text-color: #8c93a8;+ --docsearch-highlight-color: #f3f4fa;++ --color-background: #1a1b21;+ --color-background-subtle: #2a2d39;+ --color-background-muted: #252832;+ --color-field-background: #101114;+ --color-separator: #343847;+ --color-border: #414758;+ --color-card-background: #1e2027;+ --color-card-background-hover: #252832;+ --color-text: #f3f4fa;+ --color-text-subtle: #8c93a8;+ --color-text-muted: #b2b8cc;+ --color-text-on-primary: #1a1b21;+ --color-text-placeholder: #6e758a;+ --color-black-action: #fff;+ --color-icon: #b2b8cc;+ --color-hover: #2d313e;+ --color-primary-action-hover: #d1d5e4;}:root {/* use default system font based on https://devhints.io/css-system-font-stack */- --ifm-font-family-base: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;+ --ifm-font-family-base:+ -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',+ 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',+ sans-serif;--ifm-heading-font-family: 'Lota Grotesque', sans-serif;--ifm-font-weight-semibold: 600;--ifm-font-color-base: #242736;- --ifm-navbar-item-padding-horizontal: 1.75rem;+ --ifm-navbar-item-padding-horizontal: 0;+ --ifm-navbar-item-padding-vertical: 0;+--ifm-navbar-link-color: #41465d;--ifm-navbar-shadow: none;--ifm-heading-margin-top: var(--ifm-heading-margin-bottom);--ifm-hero-background-color: transparent;+ --ifm-navbar-sidebar-width: 100%;+--ifm-code-background: var(--ifm-pre-background) !important;--ifm-code-padding-horizontal: 0.4rem;--ifm-code-padding-vertical: 0.2rem;@@ -51,70 +78,89 @@ html[data-theme='dark'] {--ifm-footer-background-color: #272c3d;--ifm-footer-title-color: #f2f3fb;--ifm-footer-link-color: #f2f3fb;+ --ifm-menu-color-active: #555d76;--max-layout-width: 1680px;--docusaurus-highlighted-code-line-bg: rgba(0, 255, 77, 0.1);+ --docsearch-highlight-color: #242836;--ifm-heading-color: #242736;++ --docsearch-text-color: #6c7590;+ --docsearch-highlight-color: #242836;++ --color-background: #fff;+ --color-background-subtle: #f3f4fa;+ --color-background-muted: #f8f9fc;+ --color-field-background: #f8f9fc;+ --color-separator: #e0e3f2;+ --color-border: #d0d5e9;+ --color-card-background: #fff;+ --color-card-background-hover: #f8f9fc;+ --color-text: #242836;+ --color-text-subtle: #6c7590;+ --color-text-muted: #555d76;+ --color-text-on-primary: #fff;+ --color-text-placeholder: #969eb8;+ --color-black-action: #272d3e;+ --color-icon: #555d76;+ --color-hover: #eef0f8;+ --color-primary-action-hover: #2b3143;}@font-face {font-family: 'Lota Grotesque';- src: url('/font/lota.woff2') format('woff2'),- url('/font/lota.woff') format('woff');- font-weight: 600;-}--.footer__title {- font-size: 1.25rem;+ src:+ url('/font/lota.woff2') format('woff2'),+ url('/font/lota.woff') format('woff');font-weight: 600;}-.footer__bottom a {- opacity: 0.75;-}--.footer__copyright {- color: var(--ifm-footer-title-color);-}--.navbar__title {- /* Replaced by SVG */- display: none;-}--.navbar__inner {- /* .container */- max-width: calc(var(--max-layout-width) - 32px);- padding: 0 var(--ifm-spacing-horizontal);- width: 100%;-}--.DocSearch-Button-Placeholder {- font-size: 14px !important;+footer,+nav {+ --max-layout-width: 1200px;}html .DocSearch-Button {border-radius: 6px !important;font-weight: 400 !important;- background: #f9fafd;- border: 1px solid #c1c6dd;+ background: var(--color-field-background) !important;+ border: 1px solid var(--color-border) !important;+ width: 256px;+ height: 40px;+ padding: 0;+ padding-inline: 4px;/* Annoying, but needed *//* https://stackoverflow.com/questions/26140050/why-is-font-family-not-inherited-in-button-tags-automatically/26140154 */font-family: inherit;+ color: var(--color-text-placeholder);++ &:hover {+ color: var(--color-text-muted);+ box-shadow: none !important;+ background: var(--color-field-background) !important;+ }+}+.DocSearch-Button-Placeholder {+ display: block !important;+ font-size: 16px !important;}-html .DocSearch-Button .DocSearch-Search-Icon {- color: var(--docsearch-muted-color);+.DocSearch-Search-Icon {+ display: none;+}++div[class*='navbarSearchContainer'] {+ position: static;}-html[data-theme="dark"] .DocSearch-Button {+html[data-theme='dark'] .DocSearch-Button {background: none;border: 1px solid var(--docsearch-muted-color);}-html[data-theme="dark"] .DocSearch-Button .DocSearch-Search-Icon {+html[data-theme='dark'] .DocSearch-Button .DocSearch-Search-Icon {color: var(--docsearch-muted-color);}@@ -122,7 +168,8 @@ html[data-theme="dark"] .DocSearch-Button .DocSearch-Search-Icon {box-shadow: none !important;}-.navbar, .main-wrapper {+.navbar,+.main-wrapper {justify-content: center;}@@ -142,21 +189,123 @@ aside > div > a > b {display: none;}+@media (max-width: 1200px) {+ .navbar__toggle {+ display: inherit;+ }+ .navbar__item {+ display: none;+ }+}++@media (max-width: 767px) {+ .navbar__items--right > div,+ .navbar__items--right > a {+ display: none;+ }+}++.navbar__toggle {+ margin: 0;+ padding: 8px !important;++ svg {+ color: var(--color-icon);+ width: 20px;+ height: 20px;+ }+}++.navbar__title {+ /* Replaced by SVG */+ display: none;+}++.navbar__inner {+ /* .container */+ max-width: var(--max-layout-width);+ margin: auto;+ width: 100%;+}++.navbar__items {+ height: 28px;+ @media (min-width: 768px) {+ height: 40px;+ }+}++.navbar__items--right {+ gap: 16px;+}++.navbar__item,+.navbar__link {+ font-size: 16px;+ font-weight: 500;+ line-height: 24px; /* 150% */+ color: var(--color-text);+ border-radius: 12px;++ &:hover,+ &:focus {+ color: var(--color-text-muted);+ background: var(--color-background-muted);+ }+}++.navbar__item {+ padding: 4px 8px;+}++.navbar__item.dropdown {+ padding: 4px 16px 4px 8px;+ a {+ display: inline-flex;+ }+}++.navbar__link--active {+ color: var(--color-text-muted);+ background: var(--color-background-muted);+}+.dropdown > .navbar__link::after {border-color: currentColor;border-style: solid;border-width: 0.1em 0.1em 0 0;content: '';display: inline-block;- height: 0.4em;+ height: 0.3em;left: 0.3em;position: relative;vertical-align: top;- width: 0.4em;- top: 9px;+ width: 0.3em;+ top: 8px;transform: rotate(135deg);}+.navbar {+ border-bottom: 1px solid var(--color-separator);+ height: auto;+ background: var(--color-background);++ padding: 16px;++ @media (min-width: 768px) {+ padding: 20px 40px;+ }+ @media (min-width: 1024px) {+ padding: 20px 64px;+ }+}++nav[class*='navbarHidden'] {+ div[class*='navbarLogo'] {+ display: none;+ }+}+.navbar .icon {font-size: 0;padding: 4px;@@ -172,28 +321,28 @@ aside > div > a > b {background-size: cover;}-.navbar .icon[href*=github]::before {+.navbar .icon[href*='github']::before {background-image: url('/static/img/github-brand.svg');}-html[data-theme="dark"] .navbar .navbar__link[href*=github]:before {+html[data-theme='dark'] .navbar .navbar__link[href*='github']:before {background-image: url('/static/img/github-brand-dark.svg');}-.navbar .icon[href*=discord]::before {+.navbar .icon[href*='discord']::before {background-image: url('/static/img/discord-brand.svg');}-html[data-theme="dark"] .navbar .navbar__link[href*=discord]:before {+html[data-theme='dark'] .navbar .navbar__link[href*='discord']:before {background-image: url('/static/img/discord-brand-dark.svg');}-.navbar svg[class*=iconExternalLink],-aside svg[class*=iconExternalLink] {+.navbar svg[class*='iconExternalLink'],+aside svg[class*='iconExternalLink'] {display: none;}-header.hero div[class^=heroButtons] {+header.hero div[class^='heroButtons'] {justify-content: inherit;}@@ -226,8 +375,26 @@ article .card h2 {}nav.navbar .dropdown__menu {+ top: 32px;min-width: 6rem;- margin-left: 20px;+ background: var(--color-card-background);+ border: 1px solid var(--color-border);+}++.dropdown__menu .dropdown__link {+ width: 100%;+ border-radius: 8px;+}++.dropdown__menu .dropdown__link--active {+ color: var(--color-text-muted);+ background: var(--color-background-muted);+}++.dropdown__menu .dropdown__link:hover,+.dropdown__menu .dropdown__link--active:hover {+ background: var(--color-background-muted);+ color: var(--color-text-muted);}.navbar__logo {@@ -243,13 +410,20 @@ nav.navbar .dropdown__menu {}.main-wrapper a[class*='sidebarLogo'] img {- height: 3rem;+ height: 28px;+ margin-top: 4px;+ margin-bottom: 24px;+ margin-left: 24px;}.main-wrapper a[class*='sidebarLogo'] b {display: none;}+div[class*='sidebarViewport'] {+ top: 22px;+}+html.plugin-pages {font-size: 18px;line-height: 32px;@@ -257,7 +431,12 @@ html.plugin-pages {html.plugin-pages h2 {font-size: 36px;- line-height: 48px;+ line-height: 46px;++ @media (min-width: 768px) {+ font-size: 48px;+ line-height: 56px;+ }}html.plugin-docs .theme-doc-markdown {@@ -317,6 +496,7 @@ html.plugin-docs .theme-doc-markdown h3 {.theme-doc-sidebar-menu .menu__link--active {font-weight: 700;+ color: var(--color-text-muted);}.theme-doc-sidebar-menu .menu__list-item-collapsible,@@ -329,12 +509,18 @@ html.plugin-docs .theme-doc-markdown h3 {}html[data-theme='dark'] .theme-doc-sidebar-menu .menu__link,-html[data-theme='dark'] .theme-doc-toc-desktop .table-of-contents .toc-highlight {+html[data-theme='dark']+ .theme-doc-toc-desktop+ .table-of-contents+ .toc-highlight {color: #b3b8d2;}html[data-theme='dark'] .theme-doc-sidebar-menu .menu__link--active,-html[data-theme='dark'] .theme-doc-toc-desktop .table-of-contents .table-of-contents__link--active {+html[data-theme='dark']+ .theme-doc-toc-desktop+ .table-of-contents+ .table-of-contents__link--active {color: #f2f3fb;}@@ -353,38 +539,6 @@ html[data-theme='dark'] .theme-doc-toc-desktop .table-of-contents .table-of-contdisplay: none;}-@media (min-width: 997px) and (max-width: 1250px) {- .navbar__items--right a.icon {- display: none;- }-}--@media (min-width: 997px) and (max-width: 1130px) {- .navbar__link.changelog {- display: none;- }-}--@media (max-width: 1480px) {- .navbar__item {- padding: 4px 24px;- }-}--@media (max-width: 1380px) {- .navbar__item {- padding: 4px 15px;- font-size: 16px;- }-}--@media (min-width: 997px) and (max-width: 1439px) {- footer .col--offset-9 {- --ifm-col-width: calc(4 / 12 * 100%);- margin-left: calc(8 / 12 * 100%);- }-}-html .theme-doc-sidebar-container {border: 0;}@@ -479,14 +633,15 @@ html[data-theme='dark'] .runnable-code-block svg .apify-logo {.prism-code.language-json .token-line::before,.prism-code.language-json5 .token-line::before,.prism-code.language-XML .token-line::before,-.prism-code.language-js .token-line::before {+.prism-code.language-js .token-line::before,+.prism-code.language-python .token-line::before {counter-increment: line-number;content: counter(line-number);margin-right: calc(var(--ifm-pre-padding) * 0.8);text-align: right;min-width: 1.5rem;display: inline-block;- opacity: .3;+ opacity: 0.3;left: var(--ifm-pre-padding);}@@ -500,18 +655,18 @@ html[data-theme='dark'] .runnable-code-block svg .apify-logo {border: 1px solid #b3b8d2;}-div[class^=announcementBar_] {+div[class^='announcementBar_'] {background: #d05c29;color: #fff;}-div[class^=announcementBar_] button {+div[class^='announcementBar_'] button {color: #fff;}.markdown blockquote {--ifm-alert-background-color: var(--ifm-color-info-contrast-background);- --ifm-alert-background-color-highlight: rgba(84,199,236,.15);+ --ifm-alert-background-color-highlight: rgba(84, 199, 236, 0.15);--ifm-alert-foreground-color: var(--ifm-color-info-contrast-foreground);--ifm-alert-border-color: var(--ifm-color-info-dark);--ifm-code-background: var(--ifm-alert-background-color-highlight);@@ -526,5 +681,292 @@ div[class^=announcementBar_] button {border-left-width: var(--ifm-alert-border-left-width);border-radius: var(--ifm-alert-border-radius);box-shadow: var(--ifm-alert-shadow);- padding: var(--ifm-alert-padding-vertical) var(--ifm-alert-padding-horizontal);+ padding: var(--ifm-alert-padding-vertical)+ var(--ifm-alert-padding-horizontal);+}++.DocSearch-Button-Key {+ background: var(--color-background-subtle) !important;+ box-shadow: none !important;+ border: 1px solid var(--color-border) !important;+ padding: 0 !important;+ color: var(--color-text-muted) !important;+}++.navbar-sidebar__brand {+ border-bottom: 1px solid var(--color-separator);+ flex-direction: column;+ height: auto;+ padding: 0;+}++.menu-primary {+ padding: 0;+ .menu__list-item {+ border-bottom: 1px solid var(--color-separator);+ margin: 0px 24px !important;+ a {+ margin: 8px 0px 4px;+ padding: 8px;+ }+ display: flex;+ }+ .menu__link {+ font-size: 16px;+ font-weight: 500;+ line-height: 24px;+ }+}++.navbar-sidebar__close {+ margin-left: 16px;+ svg {+ g {+ stroke: var(--color-icon);+ }+ width: 32px;+ height: 32px;+ padding: 8px;+ }+}++.DocSearch-Modal {+ font-family: var(--ifm-font-family-base);++ border-radius: 8px !important;+ border: 1px solid var(--color-border) !important;+ background: var(--color-card-background) !important;+ box-shadow: none !important;++ button {+ font-family: var(--ifm-font-family-base);+ }++ .DocSearch-Logo {+ display: none;+ }++ .DocSearch-Footer {+ flex-direction: row;+ border-top: 1px solid var(--color-border);+ background: var(--color-background);+ box-shadow: none;+ }++ .DocSearch-Label {+ color: var(--color-text-subtle);+ font-size: 14px;+ font-weight: 400;+ line-height: 20px;+ }++ .DocSearch-Commands-Key {+ border-radius: 4px;+ border: 1px solid var(--color-border);+ background: var(--color-background-subtle);+ box-shadow: none;+ g {+ stroke: var(--color-text-subtle);+ }+ }++ .DocSearch-Form {+ border-radius: 6px;+ border-radius: var(--Radius-6, 6px);+ border: 1px solid var(--color-border);+ background: var(--color-background);+ box-shadow: none;+ height: 40px;+ padding: 8px 12px;+ flex-direction: row-reverse;+ }++ .DocSearch-Input {+ color: var(--color-text);+ font-size: 14px;+ line-height: 20px;+ padding: 0;+ }++ .DocSearch-Input::placeholder {+ color: var(--color-text-placeholder);+ font-style: italic;+ }++ .DocSearch-Search-Icon {+ width: 16px;+ height: 16px;+ path {+ stroke: var(--color-text-muted);+ }+ }++ .DocSearch-Reset {+ display: none;+ }++ .DocSearch-Help {+ color: var(--color-text-subtle);+ }++ .DocSearch-Hit-source {+ color: var(--color-text-subtle);+ font-size: 14px;+ font-weight: 400;+ line-height: 20px;+ padding-bottom: 4px;+ padding-left: 12px;+ background: var(--color-card-background);+ }++ .DocSearch-Hit {+ background: transparent;+ a {+ background: transparent !important;+ padding: 0;+ box-shadow: none;+ }+ a:hover {+ background: var(--color-hover) !important;+ }+ }++ .DocSearch-Hit[aria-selected='true'] a {+ background: var(--color-hover) !important;+ }++ .DocSearch-Hit-Container {+ background: transparent;+ height: 50px;+ }++ .DocSearch-Screen-Icon {+ display: none;+ }++ .DocSearch-NoResults {+ margin: 0;+ display: flex;+ flex-direction: column;+ width: 100%;+ padding: 16px 8px;+ gap: 24px;++ .DocSearch-Title {+ color: var(--color-text);+ font-size: 16px;+ font-weight: 500;+ line-height: 24px;+ width: fit-content;+ margin: 0;+ }+ }++ .DocSearch-Hit[aria-selected='true'] .DocSearch-Hit-title,+ .DocSearch-Hit-title {+ color: var(--color-text) !important;+ font-size: 16px;+ font-style: normal;+ font-weight: 500;+ line-height: 24px; /* 150% */+ }++ .DocSearch-Hit[aria-selected='true'] .DocSearch-Hit-path,+ .DocSearch-Hit-path,+ .DocSearch-Hit[aria-selected='true'] .DocSearch-Hit-action,+ .DocSearch-Hit-action,+ .DocSearch-Hit[aria-selected='true'] .DocSearch-Hit-icon,+ .DocSearch-Hit-icon,+ .DocSearch-Hit[aria-selected='true'] .DocSearch-Hit-Tree,+ .DocSearch-Hit-Tree {+ color: var(--color-text-muted) !important;+ }++ .DocSearch-Hit[aria-selected='true'] mark,+ .DocSearch-Hit mark {+ color: var(--color-text-subtle) !important;+ }++ .DocSearch-Help {+ color: var(--color-text-subtle);+ font-size: 14px;+ font-weight: 400;+ line-height: 16px;+ }++ .DocSearch-NoResults-Prefill-List {+ padding: 0;+ li {+ list-style-type: none;+ margin-top: 4px;+ }+ }++ .DocSearch-Prefill {+ color: var(--color-text);+ font-size: 14px;+ font-weight: 500;+ line-height: 20px;+ &:hover {+ color: var(--color-text-subtle);+ text-decoration: none;+ }+ }++ .DocSearch-HitsFooter {+ color: var(--color-text-subtle);+ font-size: 14px;+ font-weight: 400;+ line-height: 16px;++ a {+ border: none;+ }++ a:hover {+ color: var(--color-text);+ }+ }++ .DocSearch-Hit-icon {+ margin-left: 8px;+ width: auto;+ height: auto;+ svg {+ width: 16px;+ height: 16px;+ }+ }++ li[id*='recentSearches'] {+ .DocSearch-Hit-icon {+ display: none;+ }+ }++ .DocSearch-SearchBar {+ padding: 16px 16px 8px;+ }++ .DocSearch-Hit-Select-Icon {+ display: none !important;+ }++ .DocSearch-Dropdown {+ padding: 0 8px;+ }++ .DocSearch-Cancel {+ color: var(--color-text-subtle);+ font-size: 14px;+ font-weight: 500;+ line-height: 20px;+ &:hover {+ color: var(--color-text);+ }+ }++ .DocSearch-NoResults-Prefill-List ul {+ padding: 0;+ }}diff --git a/website/src/pages/index.js b/website/src/pages/index.jsindex 155daa672..490be21d5 100644--- a/website/src/pages/index.js+++ b/website/src/pages/index.js@@ -1,102 +1,39 @@/* eslint-disable max-len */-import Link from '@docusaurus/Link';+import Head from '@docusaurus/Head';import useDocusaurusContext from '@docusaurus/useDocusaurusContext';-import Admonition from '@theme/Admonition';import CodeBlock from '@theme/CodeBlock';import Layout from '@theme/Layout';-import clsx from 'clsx';-import React from 'react';-import LiteYouTubeEmbed from 'react-lite-youtube-embed';+import React, { useState } from 'react';import styles from './index.module.css';-import Highlights from '../components/Highlights';-import 'react-lite-youtube-embed/dist/LiteYouTubeEmbed.css';+import HomepageCliExample from '../components/Homepage/HomepageCliExample';+import HomepageCtaSection from '../components/Homepage/HomepageCtaSection';+import HomepageHeroSection from '../components/Homepage/HomepageHeroSection';+import LanguageInfoWidget from '../components/Homepage/LanguageInfoWidget';+import LanguageSwitch from '../components/Homepage/LanguageSwitch';import RunnableCodeBlock from '../components/RunnableCodeBlock';-function Hero() {+function LanguageGetStartedSection() {return (- <header className={clsx('container', styles.heroBanner)}>- <div className="row padding-horiz--md">- <div className="col col--7">- <div className={clsx(styles.relative, 'row')}>- <div className="col">- <h1 className={styles.tagline}>- Crawlee is a web<br /> scraping and browser<br /> automation library- </h1>- <h1 className={styles.tagline}>- Crawlee is a <span className={styles.taglineGap}>web<br /> scraping</span> and <span className={styles.taglineGap}>browser<br /> automation</span> library- </h1>- </div>- </div>- <div className="row">- <div className="col">- <h2>It helps you build reliable crawlers. Fast.</h2>- </div>- </div>- <div className="row">- <div className="col">- <div className={styles.heroButtons}>- <Link to="docs/introduction" className={styles.getStarted}>Get Started</Link>- <iframe src="https://ghbtns.com/github-btn.html?user=apify&repo=crawlee&type=star&count=true&size=large" frameBorder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe>- </div>- </div>- </div>- </div>- <div className={clsx(styles.relative, 'col', 'col--5')}>- <div className={styles.logoBlur}>- <img src={require('../../static/img/logo-blur.png').default} className={clsx(styles.hideSmall)} />- </div>- <div className={styles.codeBlock}>- <CodeBlock className="language-bash">- npx crawlee create my-crawler- </CodeBlock>- </div>- </div>- </div>- </header>- );-}--function Features() {- return (- <section className={clsx('container', styles.features)}>- <div className="row">- <div className="col col--6">- <h2>Reliable crawling 🏗️</h2>- <p>- Crawlee won't fix broken selectors for you (yet), but it helps you <b>build and maintain your crawlers faster</b>.- </p>- <p>- When a website adds <a href="https://crawlee.dev/docs/guides/javascript-rendering">JavaScript rendering</a>, you don't have to rewrite everything, only switch to- one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back.- </p>- <p>- It keeps your proxies healthy by rotating them smartly with good fingerprints that make your crawlers- look human-like. It's not unblockable,- but <a href="https://blog.apify.com/daltix-python-vs-apify-sdk/" target="_blank" rel="noreferrer"><b>it will save you money in the long run</b></a>.- </p>- <p>- Crawlee is built by people who scrape for a living and use it every day to scrape millions of pages.- <a href="https://discord.com/invite/jyEM2PRvMU" target="_blank" rel="noreferrer"><b> Meet our community on Discord</b></a>.- </p>- </div>- <div className="col col--6">- <div className="video-container">- <LiteYouTubeEmbed- id="g1Ll9OlFwEQ"- params="autoplay=1&autohide=1&showinfo=0&rel=0"- title="Crawlee, the web scraping and browser automation library"- poster="maxresdefault"- webp- />- </div>- </div>- </div>+ <section className={styles.languageGetStartedSection}>+ <LanguageInfoWidget+ language="JavaScript"+ command="npx crawlee create my-crawler"+ githubUrl="https://github.com/apify/crawlee"+ to="/js"+ />+ <div className={styles.dashedSeparatorVertical} />+ <LanguageInfoWidget+ language="Python"+ command="pipx run crawlee create my-crawler"+ githubUrl="https://github.com/apify/crawlee-python"+ to="https://crawlee.dev/python"+ /></section>);}-const example = `import { PlaywrightCrawler } from 'crawlee';+const jsExample = `import { PlaywrightCrawler } from 'crawlee';// PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.const crawler = new PlaywrightCrawler({@@ -130,126 +67,120 @@ const data = await crawler.getData();console.table(data.items);`;-function ActorExample() {- return (- <section className={clsx(styles.try, 'container')}>- <div className="col">- <h2>Try Crawlee out 👾</h2>- <Admonition type="caution" title="before you start">- Crawlee requires <a href="https://nodejs.org/en/" target="_blank" rel="noreferrer"><b>Node.js 16 or higher</b></a>.- </Admonition>- <p>- The fastest way to try Crawlee out is to use the <b>Crawlee CLI</b> and choose- the <b><a href="https://crawlee.dev/docs/quick-start">Getting started</a> example</b>.- The CLI will install all the necessary dependencies and add boilerplate code for you to play with.- </p>- <CodeBlock className="language-bash">- npx crawlee create my-crawler- </CodeBlock>- <p>- If you prefer adding Crawlee <b>into your own project</b>, try the example below.- Because it uses <code>PlaywrightCrawler</code> we also need to install Playwright.- It's not bundled with Crawlee to reduce install size.- </p>- <CodeBlock className="language-bash">- npm install crawlee playwright- </CodeBlock>- <RunnableCodeBlock className="language-typescript" type="playwright">- {{- code: example,- hash: 'eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIENyYXdsZXIgc2V0dXAgZnJvbSB0aGUgcHJldmlvdXMgZXhhbXBsZS5cXG5jb25zdCBjcmF3bGVyID0gbmV3IFBsYXl3cmlnaHRDcmF3bGVyKHtcXG4gICAgLy8gVXNlIHRoZSByZXF1ZXN0SGFuZGxlciB0byBwcm9jZXNzIGVhY2ggb2YgdGhlIGNyYXdsZWQgcGFnZXMuXFxuICAgIGFzeW5jIHJlcXVlc3RIYW5kbGVyKHsgcmVxdWVzdCwgcGFnZSwgZW5xdWV1ZUxpbmtzLCBwdXNoRGF0YSwgbG9nIH0pIHtcXG4gICAgICAgIGNvbnN0IHRpdGxlID0gYXdhaXQgcGFnZS50aXRsZSgpO1xcbiAgICAgICAgbG9nLmluZm8oYFRpdGxlIG9mICR7cmVxdWVzdC5sb2FkZWRVcmx9IGlzICcke3RpdGxlfSdgKTtcXG5cXG4gICAgICAgIC8vIFNhdmUgcmVzdWx0cyBhcyBKU09OIHRvIC4vc3RvcmFnZS9kYXRhc2V0cy9kZWZhdWx0XFxuICAgICAgICBhd2FpdCBwdXNoRGF0YSh7IHRpdGxlLCB1cmw6IHJlcXVlc3QubG9hZGVkVXJsIH0pO1xcblxcbiAgICAgICAgLy8gRXh0cmFjdCBsaW5rcyBmcm9tIHRoZSBjdXJyZW50IHBhZ2VcXG4gICAgICAgIC8vIGFuZCBhZGQgdGhlbSB0byB0aGUgY3Jhd2xpbmcgcXVldWUuXFxuICAgICAgICBhd2FpdCBlbnF1ZXVlTGlua3MoKTtcXG4gICAgfSxcXG5cXG4gICAgLy8gVW5jb21tZW50IHRoaXMgb3B0aW9uIHRvIHNlZSB0aGUgYnJvd3NlciB3aW5kb3cuXFxuICAgIC8vIGhlYWRsZXNzOiBmYWxzZSxcXG5cXG4gICAgLy8gQ29tbWVudCB0aGlzIG9wdGlvbiB0byBzY3JhcGUgdGhlIGZ1bGwgd2Vic2l0ZS5cXG4gICAgbWF4UmVxdWVzdHNQZXJDcmF3bDogMjAsXFxufSk7XFxuXFxuLy8gQWRkIGZpcnN0IFVSTCB0byB0aGUgcXVldWUgYW5kIHN0YXJ0IHRoZSBjcmF3bC5cXG5hd2FpdCBjcmF3bGVyLnJ1bihbJ2h0dHBzOi8vY3Jhd2xlZS5kZXYnXSk7XFxuXFxuLy8gRXhwb3J0IHRoZSBlbnRpcmV0eSBvZiB0aGUgZGF0YXNldCB0byBhIHNpbmdsZSBmaWxlIGluXFxuLy8gLi9zdG9yYWdlL2tleV92YWx1ZV9zdG9yZXMvcmVzdWx0LmNzdlxcbmNvbnN0IGRhdGFzZXQgPSBhd2FpdCBjcmF3bGVyLmdldERhdGFzZXQoKTtcXG5hd2FpdCBkYXRhc2V0LmV4cG9ydFRvQ1NWKCdyZXN1bHQnKTtcXG5cXG4vLyBPciB3b3JrIHdpdGggdGhlIGRhdGEgZGlyZWN0bHkuXFxuY29uc3QgZGF0YSA9IGF3YWl0IGNyYXdsZXIuZ2V0RGF0YSgpO1xcbmNvbnNvbGUudGFibGUoZGF0YS5pdGVtcyk7XFxuXCJcbn0iLCJvcHRpb25zIjp7ImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbjsgY2hhcnNldD11dGYtOCIsIm1lbW9yeSI6NDA5Nn19.WKB14SjgTceKYyhONw2oXTkiOao6X4-UAS7cIuwqGvo',- }}- </RunnableCodeBlock>- </div>- </section>- );-}+const pythonExample = `import asyncio-const npmInstall = `npm install apify-npm install -g apify-cli`;-const exampleActor = `import { PlaywrightCrawler, Dataset } from 'crawlee';+from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext-// Import the \`Actor\` class from the Apify SDK.-import { Actor } from 'apify';-// Set up the integration to Apify.-await Actor.init();+async def main() -> None:+ crawler = PlaywrightCrawler(+ max_requests_per_crawl=5, # Limit the crawl to 5 requests at most.+ headless=False, # Show the browser window.+ browser_type='firefox', # Use the Firefox browser.+ )-// Crawler setup from the previous example.-const crawler = new PlaywrightCrawler({- // ...-});-await crawler.run(['https://crawlee.dev']);+ # Define the default request handler, which will be called for every request.+ @crawler.router.default_handler+ async def request_handler(context: PlaywrightCrawlingContext) -> None:+ context.log.info(f'Processing {context.request.url} ...')-// Once finished, clean up the environment.-await Actor.exit();-`;+ # Extract and enqueue all links found on the page.+ await context.enqueue_links()++ # Extract data from the page using Playwright API.+ data = {+ 'url': context.request.url,+ 'title': await context.page.title(),+ 'content': (await context.page.content())[:100],+ }++ # Push the extracted data to the default dataset.+ await context.push_data(data)++ # Run the crawler with the initial list of URLs.+ await crawler.run(['https://crawlee.dev'])++ # Export the entire dataset to a JSON file.+ await crawler.export_data('results.json')-const apifyPush = `apify login # so the CLI knows you-apify init # and the Apify platform understands your project-apify push # time to ship it!`;+ # Or work with the data directly.+ data = await crawler.get_data()+ crawler.log.info(f'Extracted data: {data.items}')-function Deployment() {++if __name__ == '__main__':+ asyncio.run(main())`;++function CodeExampleSection() {+ const [activeOption, setActiveOption] = useState('JavaScript');return (- <section className={clsx(styles.try, 'container')}>- <div className="col">- <h2>Deploy to the cloud ☁️</h2>- <p>- Crawlee is developed by <a href="https://apify.com" rel="dofollow noreferrer" target="_blank"><b>Apify</b></a>, the web scraping and automation platform.- You can deploy a <b>Crawlee</b> project wherever you want (see our deployment guides for <a href="https://crawlee.dev/docs/deployment/aws-cheerio"><b>AWS- Lambda</b></a> and <a href="https://crawlee.dev/docs/deployment/gcp-cheerio"><b>Google Cloud</b></a>), but using the - <a href="https://console.apify.com/" target="_blank" rel="noreferrer"><b>Apify platform</b></a> will give you the best experience. With a few simple steps,- you can convert your <b>Crawlee</b> project into a so-called <b>Actor</b>. Actors are serverless micro-apps that are easy to develop, run,- share, and integrate. The infra, proxies, and storages are ready to go. <a href="https://apify.com/actors" target="_blank" rel="noreferrer">Learn more about Actors</a>.- </p>- <p>- 1️⃣ First, install the <b>Apify SDK</b> to your project, as well as the <b>Apify CLI</b>. The SDK will help with the Apify integration,- while the CLI will help us with the initialization and deployment.- </p>- <CodeBlock className="language-bash">- {npmInstall}- </CodeBlock>- <p>- 2️⃣ The next step is to add <code>Actor.init()</code> to the beginning of your main script and <code>Actor.exit()</code> to the end of it.- This will enable the integration to the Apify Platform, so the <a href="https://apify.com/storage" target="_blank" rel="noreferrer">cloud- storages</a> (e.g. <code>RequestQueue</code>) will be used. The code should look like this:- </p>- <RunnableCodeBlock className="language-typescript" type="playwright">- {{- code: exampleActor,- hash: 'eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIEltcG9ydCB0aGUgYEFjdG9yYCBjbGFzcyBmcm9tIHRoZSBBcGlmeSBTREsuXFxuaW1wb3J0IHsgQWN0b3IgfSBmcm9tICdhcGlmeSc7XFxuXFxuLy8gU2V0IHVwIHRoZSBpbnRlZ3JhdGlvbiB0byBBcGlmeS5cXG5hd2FpdCBBY3Rvci5pbml0KCk7XFxuXFxuLy8gQ3Jhd2xlciBzZXR1cCBmcm9tIHRoZSBwcmV2aW91cyBleGFtcGxlLlxcbmNvbnN0IGNyYXdsZXIgPSBuZXcgUGxheXdyaWdodENyYXdsZXIoe1xcbiAgICAvLyBVc2UgdGhlIHJlcXVlc3RIYW5kbGVyIHRvIHByb2Nlc3MgZWFjaCBvZiB0aGUgY3Jhd2xlZCBwYWdlcy5cXG4gICAgYXN5bmMgcmVxdWVzdEhhbmRsZXIoeyByZXF1ZXN0LCBwYWdlLCBlbnF1ZXVlTGlua3MsIHB1c2hEYXRhLCBsb2cgfSkge1xcbiAgICAgICAgY29uc3QgdGl0bGUgPSBhd2FpdCBwYWdlLnRpdGxlKCk7XFxuICAgICAgICBsb2cuaW5mbyhgVGl0bGUgb2YgJHtyZXF1ZXN0LmxvYWRlZFVybH0gaXMgJyR7dGl0bGV9J2ApO1xcblxcbiAgICAgICAgLy8gU2F2ZSByZXN1bHRzIGFzIEpTT04gdG8gLi9zdG9yYWdlL2RhdGFzZXRzL2RlZmF1bHRcXG4gICAgICAgIGF3YWl0IHB1c2hEYXRhKHsgdGl0bGUsIHVybDogcmVxdWVzdC5sb2FkZWRVcmwgfSk7XFxuXFxuICAgICAgICAvLyBFeHRyYWN0IGxpbmtzIGZyb20gdGhlIGN1cnJlbnQgcGFnZVxcbiAgICAgICAgLy8gYW5kIGFkZCB0aGVtIHRvIHRoZSBjcmF3bGluZyBxdWV1ZS5cXG4gICAgICAgIGF3YWl0IGVucXVldWVMaW5rcygpO1xcbiAgICB9LFxcblxcbiAgICAvLyBVbmNvbW1lbnQgdGhpcyBvcHRpb24gdG8gc2VlIHRoZSBicm93c2VyIHdpbmRvdy5cXG4gICAgLy8gaGVhZGxlc3M6IGZhbHNlLFxcblxcbiAgICAvLyBVbmNvbW1lbnQgdGhpcyBvcHRpb24gdG8gc2NyYXBlIHRoZSBmdWxsIHdlYnNpdGUuXFxuICAgIG1heFJlcXVlc3RzUGVyQ3Jhd2w6IDIwLFxcbn0pO1xcblxcbi8vIEFkZCBmaXJzdCBVUkwgdG8gdGhlIHF1ZXVlIGFuZCBzdGFydCB0aGUgY3Jhd2wuXFxuYXdhaXQgY3Jhd2xlci5ydW4oWydodHRwczovL2NyYXdsZWUuZGV2J10pO1xcblxcbi8vIEV4cG9ydCB0aGUgZW50aXJldHkgb2YgdGhlIGRhdGFzZXQgdG8gYSBzaW5nbGUgZmlsZSBpblxcbi8vIC4vc3RvcmFnZS9rZXlfdmFsdWVfc3RvcmVzL3Jlc3VsdC5jc3ZcXG5jb25zdCBkYXRhc2V0ID0gYXdhaXQgY3Jhd2xlci5nZXREYXRhc2V0KCk7XFxuYXdhaXQgZGF0YXNldC5leHBvcnRUb0NTVigncmVzdWx0Jyk7XFxuXFxuLy8gT3Igd29yayB3aXRoIHRoZSBkYXRhIGRpcmVjdGx5LlxcbmNvbnN0IGRhdGEgPSBhd2FpdCBjcmF3bGVyLmdldERhdGEoKTtcXG5jb25zb2xlLmxvZyhkYXRhLml0ZW1zLnNsaWNlKDAsIDUpKTtcXG5cXG4vLyBPbmNlIGZpbmlzaGVkLCBjbGVhbiB1cCB0aGUgZW52aXJvbm1lbnQuXFxuYXdhaXQgQWN0b3IuZXhpdCgpO1xcblwiXG59Iiwib3B0aW9ucyI6eyJjb250ZW50VHlwZSI6ImFwcGxpY2F0aW9uL2pzb247IGNoYXJzZXQ9dXRmLTgiLCJtZW1vcnkiOjQwOTZ9fQ.Te7qi0ocWNsH3ujFkgIv8AO9GQ5Wk4DZeQ9-zHTy7Vo',- }}- </RunnableCodeBlock>- <p>- 3️⃣ Then you will need to <a href="https://console.apify.com/sign-up" target="_blank" rel="noreferrer">sign up for the Apify account</a>. Once you have it,- use the Apify CLI to log in via <code>apify login</code>. The last two steps also involve the Apify CLI. Call the <code>apify- init</code> first, which will add Apify config to your project, and finally run the <code>apify push</code> to deploy it.- </p>- <CodeBlock className="language-bash">- {apifyPush}- </CodeBlock>- </div>- </section>+ <>+ <section className={styles.codeExampleSection}>+ <div+ className={styles.dashedSeparatorVertical}+ style={{ height: '46px', placeSelf: 'center' }}+ />+ <div className={styles.languageSwitchContainer}>+ <LanguageSwitch+ options={['JavaScript', 'Python']}+ onChange={(option) => setActiveOption(option)}+ />+ </div>+ <div className={styles.fadedOutSeparator} />+ <div className={styles.decorativeRow} />+ <div className={styles.codeBlockContainer}>+ {activeOption === 'JavaScript' && (+ <RunnableCodeBlock+ className={styles.codeBlock}+ type="playwright"+ language="javascript"+ >+ {{+ code: jsExample,+ hash: 'eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIENyYXdsZXIgc2V0dXAgZnJvbSB0aGUgcHJldmlvdXMgZXhhbXBsZS5cXG5jb25zdCBjcmF3bGVyID0gbmV3IFBsYXl3cmlnaHRDcmF3bGVyKHtcXG4gICAgLy8gVXNlIHRoZSByZXF1ZXN0SGFuZGxlciB0byBwcm9jZXNzIGVhY2ggb2YgdGhlIGNyYXdsZWQgcGFnZXMuXFxuICAgIGFzeW5jIHJlcXVlc3RIYW5kbGVyKHsgcmVxdWVzdCwgcGFnZSwgZW5xdWV1ZUxpbmtzLCBwdXNoRGF0YSwgbG9nIH0pIHtcXG4gICAgICAgIGNvbnN0IHRpdGxlID0gYXdhaXQgcGFnZS50aXRsZSgpO1xcbiAgICAgICAgbG9nLmluZm8oYFRpdGxlIG9mICR7cmVxdWVzdC5sb2FkZWRVcmx9IGlzICcke3RpdGxlfSdgKTtcXG5cXG4gICAgICAgIC8vIFNhdmUgcmVzdWx0cyBhcyBKU09OIHRvIC4vc3RvcmFnZS9kYXRhc2V0cy9kZWZhdWx0XFxuICAgICAgICBhd2FpdCBwdXNoRGF0YSh7IHRpdGxlLCB1cmw6IHJlcXVlc3QubG9hZGVkVXJsIH0pO1xcblxcbiAgICAgICAgLy8gRXh0cmFjdCBsaW5rcyBmcm9tIHRoZSBjdXJyZW50IHBhZ2VcXG4gICAgICAgIC8vIGFuZCBhZGQgdGhlbSB0byB0aGUgY3Jhd2xpbmcgcXVldWUuXFxuICAgICAgICBhd2FpdCBlbnF1ZXVlTGlua3MoKTtcXG4gICAgfSxcXG5cXG4gICAgLy8gVW5jb21tZW50IHRoaXMgb3B0aW9uIHRvIHNlZSB0aGUgYnJvd3NlciB3aW5kb3cuXFxuICAgIC8vIGhlYWRsZXNzOiBmYWxzZSxcXG5cXG4gICAgLy8gQ29tbWVudCB0aGlzIG9wdGlvbiB0byBzY3JhcGUgdGhlIGZ1bGwgd2Vic2l0ZS5cXG4gICAgbWF4UmVxdWVzdHNQZXJDcmF3bDogMjAsXFxufSk7XFxuXFxuLy8gQWRkIGZpcnN0IFVSTCB0byB0aGUgcXVldWUgYW5kIHN0YXJ0IHRoZSBjcmF3bC5cXG5hd2FpdCBjcmF3bGVyLnJ1bihbJ2h0dHBzOi8vY3Jhd2xlZS5kZXYnXSk7XFxuXFxuLy8gRXhwb3J0IHRoZSBlbnRpcmV0eSBvZiB0aGUgZGF0YXNldCB0byBhIHNpbmdsZSBmaWxlIGluXFxuLy8gLi9zdG9yYWdlL2tleV92YWx1ZV9zdG9yZXMvcmVzdWx0LmNzdlxcbmNvbnN0IGRhdGFzZXQgPSBhd2FpdCBjcmF3bGVyLmdldERhdGFzZXQoKTtcXG5hd2FpdCBkYXRhc2V0LmV4cG9ydFRvQ1NWKCdyZXN1bHQnKTtcXG5cXG4vLyBPciB3b3JrIHdpdGggdGhlIGRhdGEgZGlyZWN0bHkuXFxuY29uc3QgZGF0YSA9IGF3YWl0IGNyYXdsZXIuZ2V0RGF0YSgpO1xcbmNvbnNvbGUudGFibGUoZGF0YS5pdGVtcyk7XFxuXCJcbn0iLCJvcHRpb25zIjp7ImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbjsgY2hhcnNldD11dGYtOCIsIm1lbW9yeSI6NDA5Nn19.WKB14SjgTceKYyhONw2oXTkiOao6X4-UAS7cIuwqGvo',+ }}+ </RunnableCodeBlock>+ )}+ {activeOption === 'Python' && (+ <CodeBlock language="python">{pythonExample}</CodeBlock>+ )}+ </div>+ <div className={styles.dashedSeparator} />+ <div className={styles.decorativeRow} />+ </section>++ <HomepageCliExample+ example={+ activeOption === 'JavaScript'+ ? `npx crawlee create my-crawler`+ : `pipx run 'crawlee[cli]' create my-crawler`+ }+ />+ </>);}+const PAGE_TITLE = 'Crawlee · Build reliable crawlers. Fast.';+export default function Home() {- const SvgLogo = require('../../static/img/crawlee-logo.svg').default;const { siteConfig } = useDocusaurusContext();return (- <Layout- description={siteConfig.description}>- <Hero />- <Features />- <Highlights />- <ActorExample />- <Deployment />- <div className="container">- <div className="row">- <div className="col text--center padding-top--lg padding-bottom--xl">- <SvgLogo className={styles.bottomLogo} />- </div>+ <Layout description={siteConfig.description}>+ <Head>+ <title>{PAGE_TITLE}</title>+ </Head>+ <div id={styles.homepageContainer}>+ <HomepageHeroSection />+ <LanguageGetStartedSection />+ <div className={styles.dashedSeparator} />+ <CodeExampleSection />+ <div className={styles.dashedSeparator}>+ <div+ className={styles.dashedDecorativeCircle}+ id={styles.ctaDecorativeCircle}+ /></div>+ <HomepageCtaSection showJs showPython /></div></Layout>);diff --git a/website/src/pages/index.module.css b/website/src/pages/index.module.cssindex 7bd26776d..909913e69 100644--- a/website/src/pages/index.module.css+++ b/website/src/pages/index.module.css@@ -1,254 +1,257 @@-/**- * CSS files with the .module.css suffix will be treated as CSS modules- * and scoped locally.- */--.buttons {- display: flex;- align-items: center;- justify-content: center;-}--.tagline {- font-family: 'Lota Grotesque', sans-serif;- font-size: 64px;- font-weight: 600;- line-height: 80px;- letter-spacing: 0;- text-align: left;- transition: all var(--ifm-transition-fast);+/************* PAGE LAYOUT *************/+#homepageContainer {+ width: calc(100% - 48px) !important;+ max-width: 1200px !important;+ border-left: 1px solid var(--color-separator);+ border-right: 1px solid var(--color-separator);+ margin: 0 24px;}-.taglineGap {- color: transparent !important;-}--.relative {+.dashedSeparator {position: relative;-}--.codeBlock {- position: absolute;- top: 40%;- max-width: 400px;width: 100%;+ border-bottom: 1px dashed var(--color-separator);}-.heroBanner {- padding-top: 100px;- padding-bottom: 4rem;-}--.heroBanner h1:nth-child(1) {- background: linear-gradient(225deg, #FFB200 0.1%, #FFB100 8.15%, #FFAF02 15.6%, #FEAB04 22.55%, #FDA606 29.08%, #FCA00A 35.28%, #FB980E 41.26%, #FA9013 47.1%, #F98618 52.89%, #F77B1E 58.73%, #F56F24 64.71%, #F3632B 70.91%, #F25532 77.44%, #EF473A 84.39%, #ED3842 91.84%, #EB284B 99.9%),- #FCA000;- -webkit-background-clip: text;- -webkit-text-fill-color: transparent;- background-clip: text;+.dashedSeparatorVertical {+ position: relative;+ border-right: 1px dashed var(--color-separator);}-.heroBanner h1:nth-child(2) {+.dashedDecorativeCircle {+ width: 120px;+ height: 120px;+ border: 1px dashed var(--color-separator);+ border-radius: 50%;position: absolute;- top: 0;- z-index: 1;- width: calc(100% - 2rem);-}--.heroBanner h1::selection,-.heroBanner h1 span::selection {- color: rgb(36, 39, 54) !important;- -webkit-text-fill-color: rgb(36, 39, 54);- background: #B4D7FE !important;- -webkit-background-clip: unset;- background-clip: unset;-}+ transform: translate(-50%, -50%);+}++.fadedOutSeparator {+ border: none;+ height: 1px;+ background-image:+ linear-gradient(+ 90deg,+ transparent,+ transparent 50%,+ var(--color-background) 50%,+ var(--color-background) 100%+ ),+ linear-gradient(+ 90deg,+ var(--color-separator) 0%,+ transparent 50%,+ var(--color-separator) 100%+ );+ background-size:+ 6px 1px,+ 100% 1px;+}++.fadedOutSeparatorVertical {+ border: none;+ width: 1px;+ background-image:+ linear-gradient(+ 180deg,+ transparent,+ transparent 50%,+ var(--color-background) 50%,+ var(--color-background) 100%+ ),+ linear-gradient(+ 180deg,+ var(--color-separator) 0%,+ transparent 50%,+ var(--color-separator) 100%+ );+ background-size:+ 1px 6px,+ 1px 100%;+}++.trianglesSeparator {+ width: 100%;+ height: 32px;+ background-position: center;+ background-repeat: repeat-x;+ background-image: url('../../static/img/triangles_light.svg');-html[data-theme='dark'] .heroBanner ::selection {- color: #fff !important;- -webkit-text-fill-color: #fff;- background: #385477 !important;-}+ html[data-theme='dark'] & {+ background-image: url('../../static/img/triangles_dark.svg');+ }-html .heroBanner h2 {- font-style: normal;- font-weight: 400;- font-size: 24px;- line-height: 40px;- color: #41465d;- margin-top: 8px;- margin-bottom: 24px;- font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;+ /* TABLET */+ @media (min-width: 768px) {+ background-position: unset;+ background-repeat: repeat;+ height: 52px;+ }}-html[data-theme='dark'] .heroBanner h2 {- color: #b3b8d2;+/* most separators and decorations are not displayed on mobile */+.dashedSeparatorVertical,+.dashedDecorativeCircle,+.fadedOutSeparator,+.fadedOutSeparatorVertical {+ display: none;}-.heroBanner code {- background: #272c3d;- padding: 10px 20px;-}+/* TABLET */+@media (min-width: 768px) {+ .dashedSeparatorVertical,+ .dashedDecorativeCircle,+ .fadedOutSeparator,+ .fadedOutSeparatorVertical {+ display: block;+ }-.heroBanner button {- opacity: 0.4;- padding: 5px 8px;- margin-top: -2px;+ #homepageContainer {+ width: calc(100% - 80px) !important;+ margin: 0 40px;+ }}-.heroBanner button span {- width: 16px;- height: 16px;- padding: 0;- margin: 0;-}+/* DESKTOP */+@media (min-width: 1024px) {+ .dashedSeparatorVertical,+ .dashedDecorativeCircle,+ .fadedOutSeparator,+ .fadedOutSeparatorVertical {+ display: block;+ }-.heroBanner code span {- color: #f2f3fb;+ #homepageContainer {+ width: calc(100% - 128px) !important;+ margin: 0 64px;+ }}-.logoBlur {- position: absolute;- width: 680px;- height: 680px;- top: -120px;- left: -100px;- z-index: -1;-}+/************* LANGUAGE GET STARTED SECTION *************/-.heroButtons {+.languageGetStartedSection {display: flex;- align-items: center;- gap: 1rem;- margin-bottom: 50px;-}+ flex-direction: column;+ gap: 32px;+ margin: 0 0 24px 0;-@media only screen and (min-device-width: 1101px) and (max-device-width: 1440px) {- .tagline {- font-size: 56px;- line-height: 78px;+ div[class^='languageGetStartedContainer'] {+ flex: 1;}}-@media only screen and (min-device-width: 997px) and (max-device-width: 1100px) {- .tagline {- font-size: 48px;- line-height: 64px;+/* TABLET */+@media (min-width: 768px) {+ .languageGetStartedSection {+ flex-direction: row;+ align-items: stretch;+ justify-content: space-around;+ gap: 0;+ margin: 0;}}-@media only screen and (max-device-width: 996px) {- .codeBlock {- position: relative;- top: 50px;- }+/************* CODE EXAMPLE SECTION *************/- .logoBlur {- display: none;+.codeExampleSection {+ display: flex;+ flex-direction: column;+}++.languageSwitchContainer {+ place-self: center;+ margin: 32px 0 16px 0;+}++.codeBlockContainer {+ :global(.theme-code-block) {+ margin-bottom: 32px;+ border-radius: 0;+ box-shadow: none;+ border-bottom: 1px dashed var(--color-separator);+ border-top: 1px dashed var(--color-separator);++ code {+ font-size: 14px;+ background: var(--color-background-muted);+ padding: 16px 8px 16px 4px;+ span::before {+ margin-right: 16px !important;+ color: var(--color-text-subtle);+ opacity: 1 !important;+ }+ }+ pre {+ border-radius: 0;+ }}}-@media only screen and (max-device-width: 736px) {- .heroBanner {- padding-top: 20px;- padding-bottom: 2rem;+/* TABLET */+@media (min-width: 768px) {+ .codeBlockContainer :global(.theme-code-block) {+ margin-bottom: 0;+ border-bottom: none;+ border-top: none;}-- .tagline {- font-size: 32px;- line-height: 48px;+ .codeExampleSection {+ position: relative;}- .tagline br {- display: none;+ .languageSwitchContainer {+ margin: 0;+ position: absolute;+ top: calc(46px - 18px);+ left: calc(50% - 90px);}- .hideSmall {- display: none;+ .decorativeRow {+ position: relative;+ height: 46px;+ border-bottom: 1px dashed var(--color-separator);++ &::before {+ content: ' ';+ position: absolute;+ left: 40px;+ height: 100%;+ border-right: 1px dashed var(--color-separator);+ }++ &::after {+ content: ' ';+ position: absolute;+ right: 40px;+ height: 100%;+ border-left: 1px dashed var(--color-separator);+ }}- .codeBlock {- top: 0;+ .codeBlockContainer {+ margin: 0 40px;+ border-left: 1px dashed var(--color-separator);+ border-right: 1px dashed var(--color-separator);}}-@media only screen and (max-device-width: 450px) {- .codeBlock code {- font-size: 0.8em;- }+@media (min-width: 1024px) {+ .decorativeRow {+ &::before {+ left: 60px;+ }- .heroButtons {- flex-direction: column;- align-items: flex-start !important;+ &::after {+ right: 60px;+ }}-- .heroBanner button {- opacity: 0;+ .codeBlockContainer {+ margin: 0 60px;}}-@media only screen and (max-device-width: 350px) {- .codeBlock code {- font-size: 0.7em;- }-}--.tagline span {- color: var(--ifm-color-primary);-}--.getStarted {- font-size: 18px;- line-height: 28px;- padding: 12px 24px;- background: #ed3545;- border-radius: 8px;- color: white;- font-weight: 600;-}--.getStarted:hover {- color: white;- background: #ad1934;-}--html[data-theme='dark'] .getStarted {- border-color: #585e76;-}--.try {- padding-top: 20px;-}--.try, .features {- color: #41465d;-}--html[data-theme='dark'] .try,-html[data-theme='dark'] .features {- color: #b3b8d2;-}--.features > * {- margin: 2em 0;-}--.bottomLogo path:nth-child(1) {- fill: url(#gradient-1) !important;- stroke: url(#gradient-1) !important;-}--.crawleeForPython {- font-size: 44px;- font-weight: 700;- line-height: 60px;- text-align: center;- margin-bottom: 80px;-}--.crawleeForPython a {- background: linear-gradient(90deg, #833ab4, #fd1d1d 50%, #fcb045);- -webkit-background-clip: text;- background-clip: text;- -webkit-text-fill-color: #0000;+#ctaDecorativeCircle {+ width: 120px;+ height: 120px;}diff --git a/website/src/pages/js.js b/website/src/pages/js.jsnew file mode 100644index 000000000..3950ef491--- /dev/null+++ b/website/src/pages/js.js@@ -0,0 +1,394 @@+/* eslint-disable max-len */+import Link from '@docusaurus/Link';+import useDocusaurusContext from '@docusaurus/useDocusaurusContext';+import CodeBlock from '@theme/CodeBlock';+import Layout from '@theme/Layout';+import ThemedImage from '@theme/ThemedImage';+import clsx from 'clsx';+import React from 'react';++import commonStyles from './index.module.css';+import styles from './js.module.css';+import Button from '../components/Button';+import HomepageCliExample from '../components/Homepage/HomepageCliExample';+import HomepageCtaSection from '../components/Homepage/HomepageCtaSection';+import HomepageHeroSection from '../components/Homepage/HomepageHeroSection';+import LanguageInfoWidget from '../components/Homepage/LanguageInfoWidget';+import RiverSection from '../components/Homepage/RiverSection';+import ThreeCardsWithIcon from '../components/Homepage/ThreeCardsWithIcon';+import RunnableCodeBlock from '../components/RunnableCodeBlock';++function GetStartedSection() {+ return (+ <section className={styles.getStartedSection}>+ <LanguageInfoWidget+ language="JavaScript"+ githubUrl="https://github.com/apify/crawlee"+ to="/js/docs/quick-start"+ />+ </section>+ );+}++const example = `import { PlaywrightCrawler } from 'crawlee';++const crawler = new PlaywrightCrawler({+ async requestHandler({ request, page, enqueueLinks, pushData, log }) {+ const title = await page.title();+ log.info(\`Title of \${request.loadedUrl} is '\${title}'\`);++ await pushData({ title, url: request.loadedUrl });+ await enqueueLinks();+ },++ // Uncomment this option to see the browser window.+ // headless: false,+});++await crawler.run(['https://crawlee.dev']);+`;++function CodeExampleSection() {+ return (+ <section className={commonStyles.codeExampleSection}>+ <div className={commonStyles.decorativeRow} />+ <div className={commonStyles.codeBlockContainer}>+ <RunnableCodeBlock+ className={commonStyles.codeBlock}+ type="playwright"+ language="javascript"+ >+ {{+ code: example,+ hash: 'eyJ1IjoiRWdQdHczb2VqNlRhRHQ1cW4iLCJ2IjoxfQ.eyJpbnB1dCI6IntcbiAgICBcImNvZGVcIjogXCJpbXBvcnQgeyBQbGF5d3JpZ2h0Q3Jhd2xlciB9IGZyb20gJ2NyYXdsZWUnO1xcblxcbi8vIENyYXdsZXIgc2V0dXAgZnJvbSB0aGUgcHJldmlvdXMgZXhhbXBsZS5cXG5jb25zdCBjcmF3bGVyID0gbmV3IFBsYXl3cmlnaHRDcmF3bGVyKHtcXG4gICAgLy8gVXNlIHRoZSByZXF1ZXN0SGFuZGxlciB0byBwcm9jZXNzIGVhY2ggb2YgdGhlIGNyYXdsZWQgcGFnZXMuXFxuICAgIGFzeW5jIHJlcXVlc3RIYW5kbGVyKHsgcmVxdWVzdCwgcGFnZSwgZW5xdWV1ZUxpbmtzLCBwdXNoRGF0YSwgbG9nIH0pIHtcXG4gICAgICAgIGNvbnN0IHRpdGxlID0gYXdhaXQgcGFnZS50aXRsZSgpO1xcbiAgICAgICAgbG9nLmluZm8oYFRpdGxlIG9mICR7cmVxdWVzdC5sb2FkZWRVcmx9IGlzICcke3RpdGxlfSdgKTtcXG5cXG4gICAgICAgIC8vIFNhdmUgcmVzdWx0cyBhcyBKU09OIHRvIC4vc3RvcmFnZS9kYXRhc2V0cy9kZWZhdWx0XFxuICAgICAgICBhd2FpdCBwdXNoRGF0YSh7IHRpdGxlLCB1cmw6IHJlcXVlc3QubG9hZGVkVXJsIH0pO1xcblxcbiAgICAgICAgLy8gRXh0cmFjdCBsaW5rcyBmcm9tIHRoZSBjdXJyZW50IHBhZ2VcXG4gICAgICAgIC8vIGFuZCBhZGQgdGhlbSB0byB0aGUgY3Jhd2xpbmcgcXVldWUuXFxuICAgICAgICBhd2FpdCBlbnF1ZXVlTGlua3MoKTtcXG4gICAgfSxcXG5cXG4gICAgLy8gVW5jb21tZW50IHRoaXMgb3B0aW9uIHRvIHNlZSB0aGUgYnJvd3NlciB3aW5kb3cuXFxuICAgIC8vIGhlYWRsZXNzOiBmYWxzZSxcXG5cXG4gICAgLy8gQ29tbWVudCB0aGlzIG9wdGlvbiB0byBzY3JhcGUgdGhlIGZ1bGwgd2Vic2l0ZS5cXG4gICAgbWF4UmVxdWVzdHNQZXJDcmF3bDogMjAsXFxufSk7XFxuXFxuLy8gQWRkIGZpcnN0IFVSTCB0byB0aGUgcXVldWUgYW5kIHN0YXJ0IHRoZSBjcmF3bC5cXG5hd2FpdCBjcmF3bGVyLnJ1bihbJ2h0dHBzOi8vY3Jhd2xlZS5kZXYnXSk7XFxuXFxuLy8gRXhwb3J0IHRoZSBlbnRpcmV0eSBvZiB0aGUgZGF0YXNldCB0byBhIHNpbmdsZSBmaWxlIGluXFxuLy8gLi9zdG9yYWdlL2tleV92YWx1ZV9zdG9yZXMvcmVzdWx0LmNzdlxcbmNvbnN0IGRhdGFzZXQgPSBhd2FpdCBjcmF3bGVyLmdldERhdGFzZXQoKTtcXG5hd2FpdCBkYXRhc2V0LmV4cG9ydFRvQ1NWKCdyZXN1bHQnKTtcXG5cXG4vLyBPciB3b3JrIHdpdGggdGhlIGRhdGEgZGlyZWN0bHkuXFxuY29uc3QgZGF0YSA9IGF3YWl0IGNyYXdsZXIuZ2V0RGF0YSgpO1xcbmNvbnNvbGUudGFibGUoZGF0YS5pdGVtcyk7XFxuXCJcbn0iLCJvcHRpb25zIjp7ImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbjsgY2hhcnNldD11dGYtOCIsIm1lbW9yeSI6NDA5Nn19.WKB14SjgTceKYyhONw2oXTkiOao6X4-UAS7cIuwqGvo',+ }}+ </RunnableCodeBlock>+ </div>+ <div className={commonStyles.dashedSeparator} />+ <div className={commonStyles.decorativeRow} />+ </section>+ );+}++const benefitsCodeBlockCrawler = `{+ fingerprintOptions: {+ fingerprintGeneratorOptions: {+ browsers: ['chrome', 'firefox'],+ devices: ['mobile'],+ locales: ['en-US'],+ },+ },+},+`;++const benefitsCodeBlockHeadless = `const crawler = new AdaptivePlaywrightCrawler({+ renderingTypeDetectionRatio: 0.1,+ async requestHandler({ querySelector, enqueueLinks }) {+ // The crawler detects if JS rendering is needed+ // to extract this data. If not, it will use HTTP+ // for follow-up requests to save time and costs.+ const $prices = await querySelector('span.price')+ await enqueueLinks();+ },+});+`;++function BenefitsSection() {+ return (+ <section className={styles.benefitsSection}>+ <h2>What are the benefits?</h2>+ <RiverSection+ title="Unblock websites by default"+ description="Crawlee crawls stealthily with zero configuration, but you can customize its behavior to overcome any protection. Real-world fingerprints included."+ content={+ <CodeBlock className="code-block" language="javascript">+ {benefitsCodeBlockCrawler}+ </CodeBlock>+ }+ to="/js/docs/guides/avoid-blocking"+ />+ <div className={commonStyles.trianglesSeparator} />+ <RiverSection+ title="Work with your favorite tools"+ description="Crawlee integrates BeautifulSoup, Cheerio, Puppeteer, Playwright, and other popular open-source tools. No need to learn new syntax."+ content={+ <ThemedImage+ alt="Work with your favorite tools"+ sources={{+ light: '/img/favorite-tools-light.webp',+ dark: '/img/favorite-tools-dark.webp',+ }}+ />+ }+ reversed+ to="/js/docs/quick-start#choose-your-crawler"+ />+ <div className={commonStyles.trianglesSeparator} />+ <RiverSection+ title="One API for headless and HTTP"+ description="Switch between HTTP and headless without big rewrites thanks to a shared API. Or even let Adaptive crawler decide if JS rendering is needed."+ content={+ <CodeBlock className="code-block" language="javascript">+ {benefitsCodeBlockHeadless}+ </CodeBlock>+ }+ to="/js/api/core"+ />+ </section>+ );+}++function OtherFeaturesSection() {+ return (+ <section className={styles.otherFeaturesSection}>+ <h2>What else is in Crawlee?</h2>+ <div className={styles.cardsWithContentContainer}>+ <div className={styles.cardsWithImageContainer}>+ <Link className={styles.cardWithImage} to="/js/docs/guides/scaling-crawlers">+ <ThemedImage+ sources={{+ light: '/img/auto-scaling-light.webp',+ dark: '/img/auto-scaling-dark.webp',+ }}+ alt=""+ />+ <div className={styles.cardWithImageText}>+ <h3 className={styles.cardWithImageTitle}>+ Auto scaling+ </h3>+ <div className={styles.cardWithImageDescription}>+ Crawlers automatically adjust concurrency based+ on available system resources. Avoid memory+ errors in small containers and run faster in+ large ones.+ </div>+ </div>+ </Link>+ <Link className={styles.cardWithImage} to="/js/docs/guides/proxy-management">+ <ThemedImage+ sources={{+ light: '/img/smart-proxy-light.webp',+ dark: '/img/smart-proxy-dark.webp',+ }}+ alt=""+ />+ <div className={styles.cardWithImageText}>+ <h3 className={styles.cardWithImageTitle}>+ Smart proxy rotation+ </h3>+ <div className={styles.cardWithImageDescription}>+ Crawlee uses a pool of sessions represented by+ different proxies to maintain the proxy+ performance and keep IPs healthy. Blocked+ proxies are removed from the pool automatically.+ </div>+ </div>+ </Link>+ </div>+ <ThreeCardsWithIcon+ cards={[+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/queue-light-icon.svg',+ dark: '/img/queue-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Queue and storage',+ description:+ 'Pause and resume crawlers thanks to a persistent queue of URLs and storage for structured data.',+ to: '/js/docs/guides/request-storage',+ },+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/scraping-utils-light-icon.svg',+ dark: '/img/scraping-utils-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Handy scraping utils',+ description:+ 'Sitemaps, infinite scroll, contact extraction, large asset blocking and many more utils included.',+ to: '/js/docs/guides/avoid-blocking',+ },+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/routing-light-icon.svg',+ dark: '/img/routing-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Routing & middleware',+ description:+ 'Keep your code clean and organized while managing complex crawls with a built-in router that streamlines the process.',+ to: '/js/api/core/class/Router',+ },+ ]}+ />+ </div>+ </section>+ );+}++function DeployToCloudSection() {+ return (+ <section className={styles.deployToCloudSection}>+ <div className={styles.deployToCloudLeftSide}>+ <h2>Deploy to cloud </h2>+ <div className={styles.deployToCloudDescription}>+ Crawlee, by Apify, works anywhere, but Apify offers the best+ experience. Easily turn your project into an{' '}+ <Link to="https://apify.com/actors" rel="dofollow">+ Actor+ </Link>+ —a serverless micro-app with built-in infra, proxies, and+ storage.+ </div>+ <Button+ withIcon+ to="/js/docs/deployment/apify-platform"+ >+ Deploy to Apify+ </Button>+ </div>+ <div className={styles.deployToCloudRightSide}>+ <div+ className={commonStyles.dashedSeparatorVertical}+ id={styles.verticalStepLine}+ />+ <div className={styles.deployToCloudStep}>+ <div className={styles.deployToCloudStepNumber}>+ <div>1</div>+ </div>+ <div className={styles.deployToCloudStepText}>+ Install Apify SDK and Apify CLI.+ </div>+ </div>+ <div className={styles.deployToCloudStep}>+ <div className={styles.deployToCloudStepNumber}>+ <div>2</div>+ </div>+ <div className={styles.deployToCloudStepText}>+ Add <pre>Actor.init()</pre> to the begining and{' '}+ <pre>Actor.exit()</pre> to the end of your code.+ </div>+ </div>+ <div className={styles.deployToCloudStep}>+ <div className={styles.deployToCloudStepNumber}>+ <div>3</div>+ </div>+ <div className={styles.deployToCloudStepText}>+ Use the Apify CLI to push the code to the Apify+ platform.+ </div>+ </div>+ </div>+ </section>+ );+}++function BuildFastScrapersSection() {+ return (+ <section className={styles.buildFastScrapersSection}>+ <div className={commonStyles.dashedDecorativeCircle} />+ <div className={commonStyles.dashedSeparator} />+ <h2>Crawlee helps you build scrapers faster</h2>+ <ThreeCardsWithIcon+ cards={[+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/zero-setup-light-icon.svg',+ dark: '/img/zero-setup-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Zero setup required',+ description:+ 'Copy code example, install Crawlee and go. No CLI required, no complex file structure, no boilerplate.',+ actionLink: {+ text: 'Get started',+ href: '/js/docs/quick-start',+ },+ },+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/defaults-light-icon.svg',+ dark: '/img/defaults-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Reasonable defaults',+ description:+ 'Unblocking, proxy rotation and other core features are already turned on. But also very configurable.',+ actionLink: {+ text: 'Learn more',+ href: '/js/docs/guides/configuration',+ },+ },+ {+ icon: (+ <ThemedImage+ sources={{+ light: '/img/community-light-icon.svg',+ dark: '/img/community-dark-icon.svg',+ }}+ alt=""+ />+ ),+ title: 'Helpful community',+ description:+ 'Join our Discord community of over 10k developers and get fast answers to your web scraping questions.',+ actionLink: {+ text: 'Join Discord',+ href: 'https://discord.gg/jyEM2PRvMU',+ },+ },+ ]}+ />+ </section>+ );+}++export default function JavascriptHomepage() {+ const { siteConfig } = useDocusaurusContext();+ return (+ <Layout description={siteConfig.description}>+ <div id={commonStyles.homepageContainer}>+ <HomepageHeroSection />+ <GetStartedSection />+ <div className={clsx(commonStyles.dashedSeparator, styles.codeExampleTopSeparator)} />+ <CodeExampleSection />+ <HomepageCliExample example={`npx crawlee create my-crawler`} />+ <div className={commonStyles.dashedSeparator}>+ <div+ className={commonStyles.dashedDecorativeCircle}+ id={commonStyles.ctaDecorativeCircle}+ />+ </div>+ <BenefitsSection />+ <div className={commonStyles.dashedSeparator} />+ <OtherFeaturesSection />+ <div className={commonStyles.dashedSeparator} />+ <DeployToCloudSection />+ <div className={commonStyles.dashedSeparator} />+ <BuildFastScrapersSection />+ <HomepageCtaSection showJs />+ </div>+ </Layout>+ );+}diff --git a/website/src/pages/js.module.css b/website/src/pages/js.module.cssnew file mode 100644index 000000000..c3734c5a5--- /dev/null+++ b/website/src/pages/js.module.css@@ -0,0 +1,329 @@+.getStartedSection {+ margin-bottom: 32px;+ @media (min-width: 768px) {+ margin-bottom: 0;+ }+}++.codeExampleTopSeparator {+ display: none;+}+@media (min-width: 768px) {+ .codeExampleTopSeparator {+ display: block;+ }+}++/************** BENEFITS SECTION ***********/++.benefitsSection {+ margin-bottom: 60px;++ h2 {+ padding: 0 12px;+ margin: 32px 0;+ text-align: center;++ /* TABLET */+ @media (min-width: 768px) {+ margin: 80px 0;+ }+ }+}++/************** OTHER FEATURES SECTION ***********/++.otherFeaturesSection {+ display: flex;+ flex-direction: column;++ h2 {+ margin: 0;+ padding: 32px 12px;++ text-align: center;+ color: var(--color-text);+ font-weight: 400;++ line-height: 46px !important;+ font-size: 36px !important;++ @media (min-width: 768px) {+ line-height: 56px !important;+ font-size: 48px !important;+ margin: 80px 0 64px;+ padding: 0 24px;+ }+ }+ margin-bottom: 40px;++ @media (min-width: 768px) {+ margin-bottom: 80px;+ }+}++.cardsWithContentContainer {+ display: flex;+ flex-direction: column;+ gap: 20px;+ background-position-x: 5px;+ background-image: url('../../static/img/triangles_light.svg');++ html[data-theme='dark'] & {+ background-image: url('../../static/img/triangles_dark.svg');+ }++ @media (min-width: 768px) {+ gap: 48px;+ }+}++.cardsWithImageContainer {+ display: flex;+ flex-direction: column;+ gap: 20px;+ width: 100%;++ @media (min-width: 768px) {+ gap: 32px;+ flex-direction: row;+ }+}++.cardWithImage {+ flex: 1;+ display: flex;+ flex-direction: column;+ overflow: hidden;+ background: var(--color-card-background);+ border-block: 1px solid var(--color-separator);+ transition: background 0.1s ease;++ @media (min-width: 768px) {+ border: 1px solid var(--color-separator);+ }++ &:first-child {+ border-left: 0;+ }+ &:last-child {+ border-right: 0;+ }++ &:hover {+ background: var(--color-card-background-hover);+ }+}++.cardWithImage img {+ width: 100%;+ height: 250px;+ object-fit: cover;+}++.cardWithImage:last-child img {+ object-position: left 90%;+}++.cardWithImageText {+ padding: 40px 24px;+ border-top: 1px solid var(--color-separator);+}++.cardWithImageTitle {+ margin: 0;++ color: var(--color-text);+ font-size: 26px;+ font-style: normal;+ font-weight: 400;+ line-height: 34px;+}++.cardWithImageDescription {+ margin-top: 12px;+ color: var(--color-text-muted);+ font-family: var(--ifm-font-family-base);+ font-size: 16px;+ font-style: normal;+ font-weight: 400;+ line-height: 24px;+}++/************** DEPLOY TO CLOUD SECTION ***********/++.deployToCloudSection {+ padding: 32px 16px;+ display: flex;+ flex-direction: column;+ align-items: center;+ gap: 48px;+}++.deployToCloudLeftSide {+ display: flex;+ flex-direction: column;+ flex-basis: 50%;+ gap: 24px;+ text-align: center;+ font-style: normal;+ font-weight: 400;++ a {+ width: fit-content;+ margin: auto;+ }++ h2 {+ color: var(--color-text);+ font-family: 'Lota Grotesque';+ font-size: 38px;+ line-height: 46px;+ }+}++.deployToCloudDescription {+ color: var(--color-text-muted);+ font-size: 16px;+ line-height: 24px;+ a {+ color: inherit;+ text-decoration: underline;+ }+}++.deployToCloudRightSide {+ display: flex;+ flex-direction: column;+ gap: 24px;+ flex-basis: 50%;+ position: relative;+}++.deployToCloudStep {+ display: flex;+ flex-direction: row;+ gap: 16px;+ align-items: center;+}++.deployToCloudStepNumber {+ display: flex;+ justify-content: center;+ align-items: center;+ width: 72px;+ height: 72px;+ padding: 16px;+ border-radius: 8px;+ border: 1px solid var(--color-separator);+ background: var(--color-background);+ color: var(--color-text-muted);+ font-size: 16px;+ font-style: normal;+ font-weight: 400;+ line-height: 24px;+ z-index: 1;+ div {+ display: flex;+ justify-content: center;+ align-items: center;+ height: 40px;+ width: 40px;+ border-radius: 50%;+ border: 1px dashed var(--color-separator);+ flex-shrink: 0;+ }+}++.deployToCloudStepText {+ display: inline-flex;+ align-items: baseline;+ flex-wrap: wrap;+ gap: 4px;+ color: var(--color-text);+ font-size: 14px;+ font-style: normal;+ font-weight: 500;+ line-height: 20px;++ pre {+ margin: 0;+ padding: 0;+ background-color: transparent;+ }+}++#verticalStepLine {+ position: absolute;+ left: 36px;+ height: 100%;+ z-index: 0;+}++/* TABLET */+@media (min-width: 768px) {+ .deployToCloudSection {+ padding: 96px 40px;+ flex-direction: row;+ }+ .deployToCloudLeftSide {+ text-align: left;++ a {+ margin: 0;+ }++ h2 {+ color: var(--color-text);+ font-family: 'Lota Grotesque';+ font-size: 48px;+ line-height: 58px;+ }+ }+ .deployToCloudDescription {+ font-size: 18px;+ line-height: 28px;+ }+}++/************** BUILD SCRAPERS FAST SECTION ***********/++.buildFastScrapersSection {+ position: relative;++ padding: 40px 0 32px;++ border-bottom: 1px solid var(--color-separator);++ h2 {+ margin: 0;+ padding: 32px 12px;+ text-align: center;+ color: var(--color-text);+ font-weight: 400;++ line-height: 46px !important;+ font-size: 36px !important;++ @media (min-width: 768px) {+ line-height: 56px !important;+ font-size: 48px !important;+ padding: 80px 24px 64px;+ }+ }++ div[class*='dashedDecorativeCircle'] {+ display: none;+ }++ @media (min-width: 1024px) {+ padding: 80px 0 60px;+ div[class*='dashedDecorativeCircle'] {+ display: block;+ }+ }+}++.buildFastScrapersContent {+ border-block: 1px solid var(--color-separator);+}diff --git a/website/src/theme/ColorModeToggle/dark-mode-icon.svg b/website/src/theme/ColorModeToggle/dark-mode-icon.svgnew file mode 100644index 000000000..76f4ee4e5--- /dev/null+++ b/website/src/theme/ColorModeToggle/dark-mode-icon.svg@@ -0,0 +1,5 @@+<svg width="20" height="20" viewBox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg">+<path d="M10.0001 2.49983C10.1101 2.49983 10.2193 2.49983 10.3276 2.49983C9.25709 3.49459 8.56897 4.83303 8.38288 6.28251C8.1968 7.732 8.52453 9.20086 9.30909 10.4338C10.0937 11.6667 11.2855 12.5857 12.6773 13.0311C14.0692 13.4764 15.5731 13.42 16.9276 12.8715C16.4066 14.1252 15.555 15.214 14.4636 16.0217C13.3723 16.8294 12.0822 17.3257 10.7309 17.4577C9.37962 17.5896 8.01786 17.3523 6.79088 16.771C5.56391 16.1897 4.51774 15.2862 3.76397 14.157C3.0102 13.0277 2.5771 11.7151 2.51088 10.359C2.44466 9.00288 2.7478 7.65424 3.38795 6.45693C4.02811 5.25961 4.98128 4.25852 6.14578 3.56044C7.31028 2.86237 8.64244 2.49349 10.0001 2.49316V2.49983Z" stroke="#555D76" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M14.1667 3.33301C14.1667 3.77504 14.3423 4.19896 14.6548 4.51152C14.9674 4.82408 15.3913 4.99967 15.8333 4.99967C15.3913 4.99967 14.9674 5.17527 14.6548 5.48783C14.3423 5.80039 14.1667 6.22431 14.1667 6.66634C14.1667 6.22431 13.9911 5.80039 13.6785 5.48783C13.366 5.17527 12.942 4.99967 12.5 4.99967C12.942 4.99967 13.366 4.82408 13.6785 4.51152C13.9911 4.19896 14.1667 3.77504 14.1667 3.33301Z" stroke="#555D76" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M15.833 9.16634H17.4997M16.6663 8.33301V9.99967" stroke="#555D76" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/src/theme/ColorModeToggle/index.js b/website/src/theme/ColorModeToggle/index.jsindex 45e8efeb5..5841d36ca 100644--- a/website/src/theme/ColorModeToggle/index.js+++ b/website/src/theme/ColorModeToggle/index.js@@ -1,15 +1,16 @@-import React from 'react';-import clsx from 'clsx';-import useIsBrowser from '@docusaurus/useIsBrowser';import { translate } from '@docusaurus/Translate';-import IconLightMode from '../Icon/LightMode';-import IconDarkMode from '../Icon/DarkMode';+import useIsBrowser from '@docusaurus/useIsBrowser';+import clsx from 'clsx';+import React from 'react';++import IconDarkMode from './dark-mode-icon.svg';+import IconLightMode from './light-mode-icon.svg';import styles from './styles.module.css';function ColorModeToggle({className,value,- onChange+ onChange,}) {const isBrowser = useIsBrowser();const title = translate(@@ -34,7 +35,7 @@ function ColorModeToggle({},);return (- <div className={clsx(styles.toggle, className)}>+ <div className={className}><buttonclassName={clsx('clean-btn',@@ -46,16 +47,14 @@ function ColorModeToggle({disabled={!isBrowser}title={title}aria-label={title}>- <span>- <IconLightMode- className={clsx(styles.toggleIcon, styles.lightToggleIcon)}- width={14} height={14}- />- <IconDarkMode- className={clsx(styles.toggleIcon, styles.darkToggleIcon)}- width={14} height={14}- />- </span>+ <IconLightMode+ className={clsx(styles.toggleIcon, styles.lightToggleIcon)}++ />+ <IconDarkMode+ className={clsx(styles.toggleIcon, styles.darkToggleIcon)}+ />+ <span /></button></div>);diff --git a/website/src/theme/ColorModeToggle/light-mode-icon.svg b/website/src/theme/ColorModeToggle/light-mode-icon.svgnew file mode 100644index 000000000..3d16d9b9e--- /dev/null+++ b/website/src/theme/ColorModeToggle/light-mode-icon.svg@@ -0,0 +1,4 @@+<svg width="20" height="20" viewBox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg">+<path d="M6.66602 10.0003C6.66602 10.8844 7.0172 11.7322 7.64233 12.3573C8.26745 12.9825 9.11529 13.3337 9.99935 13.3337C10.8834 13.3337 11.7313 12.9825 12.3564 12.3573C12.9815 11.7322 13.3327 10.8844 13.3327 10.0003C13.3327 9.11627 12.9815 8.26842 12.3564 7.6433C11.7313 7.01818 10.8834 6.66699 9.99935 6.66699C9.11529 6.66699 8.26745 7.01818 7.64233 7.6433C7.0172 8.26842 6.66602 9.11627 6.66602 10.0003Z" stroke="#555D76" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M2.5 10H3.33333M10 2.5V3.33333M16.6667 10H17.5M10 16.6667V17.5M4.66667 4.66667L5.25 5.25M15.3333 4.66667L14.75 5.25M14.75 14.75L15.3333 15.3333M5.25 14.75L4.66667 15.3333" stroke="#555D76" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/src/theme/ColorModeToggle/styles.module.css b/website/src/theme/ColorModeToggle/styles.module.cssindex 3a5fd98a6..1458d767b 100644--- a/website/src/theme/ColorModeToggle/styles.module.css+++ b/website/src/theme/ColorModeToggle/styles.module.css@@ -1,51 +1,46 @@-.toggle {- padding: 3px;-}-.toggleButton {- width: 52px;- height: 26px;- background: #cfd4eb;- border-radius: 160px;+ padding: 4px;display: flex;+ gap: 4px;align-items: center;transition: all var(--ifm-transition-fast);-}--[data-theme='dark'] .toggleButton {- background: #585e76;+ position: relative;+ border-radius: 150px;+ background-color: var(--color-background-subtle);}.toggleButton span {- -webkit-tap-highlight-color: transparent;- align-items: center;- display: flex;- justify-content: center;- width: 20px;- height: 20px;+ width: 44px;+ height: 36px;border-radius: 50%;background: #fff;- vertical-align: middle;- margin: 3px;- position: relative;+ position: absolute;transition: all var(--ifm-transition-fast);left: 0;- color: #585e76;-}+ margin: 4px;-[data-theme='dark'] .toggleButton span {- background: #1a1b23;- color: #b3b8d2;- left: 25px;+ border-radius: 150px;+ background-color: var(--color-background);++ /* Light/L1 */+ box-shadow:+ 0px 0.5px 1.5px 0px rgba(63, 71, 93, 0.15),+ 0.4px 0.8px 1px -1.2px rgba(63, 71, 93, 0.14),+ 1px 2px 2.5px -2.5px rgba(63, 71, 93, 0.13);}-.toggleButton:hover span {- background: var(--ifm-color-emphasis-200);+.toggleButton svg {+ z-index: 1;+ margin: 8px 12px;+ width: 20px;+ height: 20px;+ path {+ stroke: var(--color-icon);+ }}-[data-theme='light'] .darkToggleIcon,-[data-theme='dark'] .lightToggleIcon {- display: none;+[data-theme='dark'] .toggleButton span {+ left: 48px;}.toggleButtonDisabled {diff --git a/website/src/theme/DocBreadcrumbs/Items/Home/index.js b/website/src/theme/DocBreadcrumbs/Items/Home/index.jsnew file mode 100644index 000000000..c920d2ca0--- /dev/null+++ b/website/src/theme/DocBreadcrumbs/Items/Home/index.js@@ -0,0 +1,23 @@+import Link from '@docusaurus/Link';+import { translate } from '@docusaurus/Translate';+import IconHome from '@theme/Icon/Home';+import React from 'react';++import styles from './styles.module.css';++export default function HomeBreadcrumbItem() {+ return (+ <li className="breadcrumbs__item">+ <Link+ aria-label={translate({+ id: 'theme.docs.breadcrumbs.home',+ message: 'Home page',+ description: 'The ARIA label for the home page in the breadcrumbs',+ })}+ className="breadcrumbs__link"+ href="/js">+ <IconHome className={styles.breadcrumbHomeIcon} />+ </Link>+ </li>+ );+}diff --git a/website/src/theme/DocBreadcrumbs/Items/Home/styles.module.css b/website/src/theme/DocBreadcrumbs/Items/Home/styles.module.cssnew file mode 100644index 000000000..533d76487--- /dev/null+++ b/website/src/theme/DocBreadcrumbs/Items/Home/styles.module.css@@ -0,0 +1,7 @@+.breadcrumbHomeIcon {+ position: relative;+ top: 1px;+ vertical-align: top;+ height: 1.1rem;+ width: 1.1rem;+}diff --git a/website/src/theme/Footer/LinkItem/index.js b/website/src/theme/Footer/LinkItem/index.jsnew file mode 100644index 000000000..488c6356a--- /dev/null+++ b/website/src/theme/Footer/LinkItem/index.js@@ -0,0 +1,31 @@+import isInternalUrl from '@docusaurus/isInternalUrl';+import Link from '@docusaurus/Link';+import useBaseUrl from '@docusaurus/useBaseUrl';+import clsx from 'clsx';+import React from 'react';++import styles from './index.module.css';++export default function FooterLinkItem({ item }) {+ const ExternalLinkIcon = require('../../../../static/img/external-link.svg').default;++ const { to, href, label, prependBaseUrlToHref, className, ...props } = item;+ const toUrl = useBaseUrl(to);+ const normalizedHref = useBaseUrl(href, { forcePrependBaseUrl: true });++ return (+ <Link+ className={clsx('footer__link-item', className, styles.footerLink)}+ {...(href+ ? {+ href: prependBaseUrlToHref ? normalizedHref : href,+ }+ : {+ to: toUrl,+ })}+ {...props}>+ {label}+ {href && !isInternalUrl(href) && <ExternalLinkIcon className={styles.externalLinkIcon} />}+ </Link>+ );+}diff --git a/website/src/theme/Footer/LinkItem/index.module.css b/website/src/theme/Footer/LinkItem/index.module.cssnew file mode 100644index 000000000..1212842eb--- /dev/null+++ b/website/src/theme/Footer/LinkItem/index.module.css@@ -0,0 +1,19 @@+.footerLink {+ color: var(--color-text);+ cursor: pointer;+ font-size: 14px;+ line-height: 20px;+ &:hover {+ color: var(--color-text-subtle);+ path {+ fill: var(--color-text-subtle);+ }+ }+}++.externalLinkIcon {+ margin-left: 5px;+ path {+ fill: var(--color-text);+ }+}diff --git a/website/src/theme/Footer/index.js b/website/src/theme/Footer/index.jsindex f510c5af6..de0c9252a 100644--- a/website/src/theme/Footer/index.js+++ b/website/src/theme/Footer/index.js@@ -1,22 +1,25 @@-import React from 'react';-import clsx from 'clsx';-// eslint-disable-next-line import/no-extraneous-dependencies+import Link from '@docusaurus/Link';import { useThemeConfig } from '@docusaurus/theme-common';import LinkItem from '@theme/Footer/LinkItem';+import NavbarColorModeToggle from '@theme/Navbar/ColorModeToggle';+import ThemedImage from '@theme/ThemedImage';+import clsx from 'clsx';+import React from 'react';+import styles from './index.module.css';function FooterLinksColumn({ column }) {return (- <>+ <div><div className={styles.footerTitle}>{column.title}</div>- <ul className={clsx(styles.footerItem, 'clean-list')}>+ <ul className={clsx(styles.footerList, 'clean-list')}>{column.items.map((item, i) => (- <li key={i} className="footer__item">+ <li key={i}><LinkItem item={item} /></li>))}</ul>- </>+ </div>);}@@ -26,43 +29,40 @@ function Footer() {return null;}const { links, style } = footer;- const OpenSourceIcon = require('../../../static/img/footer-open-source.svg').default;- const ApifyLogo = require('../../../static/img/footer-apify-logo.svg').default;+ const HearthIcon = require('../../../static/img/hearth.svg').default;+return (<footer className={clsx(styles.footer, style)}>- <div className="container padding-horiz--lg">- <div className="row">- <div className="col col--5">- <div className="row">- <div className="col col--6">- <FooterLinksColumn column={links[0]} />- </div>- <div className="col col--6">- <FooterLinksColumn column={links[1]} />- </div>- </div>+ <div className={styles.footerTop}>+ <div className={styles.footerTopRow}>+ <div className={styles.footerTopRowLeft}>+ <Link to="/" width="120" className={styles.footerLogo}>+ <ThemedImage+ width="120"+ alt="Docusaurus themed image"+ sources={{+ light: '/img/crawlee-light.svg',+ dark: '/img/crawlee-dark.svg',+ }}+ />+ </Link>+ <NavbarColorModeToggle /></div>- <div className="col col--7">- <div className="row">- <div className="col col--3 col--offset-9">- <FooterLinksColumn column={links[2]} />- </div>- </div>+ <div className={styles.footerTopRowRight}>+ <FooterLinksColumn column={links[0]} />+ <FooterLinksColumn column={links[1]} />+ <FooterLinksColumn column={links[2]} /></div></div>- <div className="row padding-vert--md padding-top--lg">- <div className="col padding-vert--md col--6">- <div className={styles.freeAndOpenSource}>- <OpenSourceIcon className={styles.alignMiddle} />- <span className={styles.alignMiddle}>Crawlee is free and open source</span>- </div>- </div>- <div className="col padding-vert--md col--6 text--right">- <span className={styles.builtBy}>- <span className={styles.alignMiddle}>Built by</span>- <a href="https://apify.com"><ApifyLogo className={styles.alignMiddle} /></a>- </span>+ </div>++ <div className={styles.footerBottom}>+ <div className={styles.footerBottomRow}>+ <div>+ <HearthIcon className={styles.hearthIcon} />+ Crawlee is forever free and open source</div>+ <div>© {new Date().getFullYear()} Apify</div></div></div></footer>diff --git a/website/src/theme/Footer/index.module.css b/website/src/theme/Footer/index.module.cssindex 53f8554f0..738bdabf7 100644--- a/website/src/theme/Footer/index.module.css+++ b/website/src/theme/Footer/index.module.css@@ -1,46 +1,110 @@.footer {- background: #272c3d;- padding-top: 64px;+ background: var(--color-background);+ color: var(--color-text);}-.builtBy {- color: #b3b8d2;-}+.footerBottom,+.footerTop {+ border-top: 1px solid var(--color-separator);-.builtBy svg {- margin-left: 10px;- width: 90px;- height: 24px;+ @media (min-width: 768px) {+ padding: 40px 40px;+ }+ @media (min-width: 1024px) {+ padding: 40px 64px;+ }}-.freeAndOpenSource {- color: #b3b8d2;+.footerTopRow {+ max-width: var(--max-layout-width);+ margin: auto;++ display: flex;+ flex-direction: column;+ @media (min-width: 768px) {+ flex-direction: row;+ }}-.alignMiddle {- vertical-align: middle;- display: inline-block;+.footerTopRowRight {+ flex-direction: column;+ display: flex;+ flex: 3;+ gap: 32px;+ padding: 16px 40px 40px;++ @media (min-width: 768px) {+ gap: 0;+ flex-direction: row;+ padding: 0;+ justify-content: space-between;+ }}-.freeAndOpenSource svg {- margin-right: 10px;+.footerTopRowLeft {+ display: flex;+ flex-direction: column;+ justify-content: space-between;+ flex: 2;+ padding: 32px 40px 24px;+ gap: 32px;++ border-bottom: 1px solid var(--color-separator);++ img {+ display: block !important;+ }++ @media (min-width: 768px) {+ padding: 0;+ border: 0;+ gap: 0;+ }}-.freeAndOpenSource svg path {- fill: #b3b8d2 !important;+.footerBottomRow {+ max-width: var(--max-layout-width);+ margin: auto;++ display: flex;+ flex-direction: column;+ align-items: center;+ gap: 24px;+ padding: 24px 40px;++ font-size: 14px;+ line-height: 20px;+ text-align: center;++ @media (min-width: 768px) {+ gap: 0;+ padding: 0;+ flex-direction: row;+ justify-content: space-between;+ }}-.footer .footer__item svg path {- fill: #6f7490;+.hearthIcon {+ margin-right: 8px;+ path {+ fill: var(--color-text-muted);+ }}.footerTitle {- font-family: 'Lota Grotesque', sans-serif;- font-weight: 600;font-size: 16px;- line-height: 20px;- letter-spacing: 0.1em;- text-transform: uppercase;- color: #8d92af;- margin-bottom: 20px;+ font-weight: 700;+ line-height: 24px;+}++.footerList {+ margin: 0;+ li {+ margin-top: 16px;+ height: 28px;+ }+}++.footerLogo {+ width: fit-content;}diff --git a/website/src/theme/Icon/DarkMode/index.js b/website/src/theme/Icon/DarkMode/index.jsdeleted file mode 100644index 77a6ccb53..000000000--- a/website/src/theme/Icon/DarkMode/index.js+++ /dev/null@@ -1,11 +0,0 @@-import React from 'react';--function IconDarkMode(props) {- return (- <svg viewBox="0 0 13 12" width={14} height={14} {...props}>- <path d="M10.7001 6.39501C10.6215 7.24611 10.3021 8.05721 9.77927 8.7334C9.25646 9.40959 8.55189 9.92291 7.748 10.2133C6.9441 10.5036 6.07414 10.5591 5.2399 10.3731C4.40565 10.187 3.64164 9.76728 3.03726 9.1629C2.43287 8.55851 2.01312 7.7945 1.8271 6.96026C1.64108 6.12602 1.6965 5.25605 1.98688 4.45216C2.27725 3.64826 2.79056 2.94369 3.46675 2.42088C4.14294 1.89808 4.95404 1.57866 5.80515 1.50001C5.30685 2.17414 5.06707 3.00473 5.12941 3.84071C5.19175 4.6767 5.55208 5.46254 6.14485 6.05531C6.73762 6.64808 7.52346 7.0084 8.35944 7.07074C9.19542 7.13308 10.026 6.8933 10.7001 6.39501Z" stroke="currentColor" fill="transparent" strokeLinecap="round" strokeLinejoin="round"/>- </svg>- );-}--export default React.memo(IconDarkMode);diff --git a/website/src/theme/Icon/LightMode/index.js b/website/src/theme/Icon/LightMode/index.jsdeleted file mode 100644index 69fb6eb83..000000000--- a/website/src/theme/Icon/LightMode/index.js+++ /dev/null@@ -1,28 +0,0 @@-import React from 'react';--function IconLightMode(props) {- return (- <svg viewBox="0 0 13 12" fill="none" xmlns="http://www.w3.org/2000/svg" width={14} height={14} {...props}>- <g clipPath="url(#clip0_833_8168)">- <path- d="M6.59998 8.49999C7.98069 8.49999 9.09998 7.3807 9.09998 5.99999C9.09998 4.61928 7.98069 3.49999 6.59998 3.49999C5.21926 3.49999 4.09998 4.61928 4.09998 5.99999C4.09998 7.3807 5.21926 8.49999 6.59998 8.49999Z"- stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M6.59985 0.5V1.5" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M6.59985 10.5V11.5" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M2.7099 2.11L3.4199 2.82" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M9.77991 9.17999L10.4899 9.88999" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M1.09998 6H2.09998" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M11.0999 6H12.0999" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M2.7099 9.88999L3.4199 9.17999" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- <path d="M9.77991 2.82L10.4899 2.11" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round"/>- </g>- <defs>- <clipPath id="clip0_833_8168">- <rect width="12" height="12" fill="white" transform="translate(0.599976)"/>- </clipPath>- </defs>- </svg>- );-}--export default React.memo(IconLightMode);diff --git a/website/src/theme/Logo/index.js b/website/src/theme/Logo/index.jsnew file mode 100644index 000000000..fe9100faa--- /dev/null+++ b/website/src/theme/Logo/index.js@@ -0,0 +1,36 @@+import Link from '@docusaurus/Link';+import { useLocation } from '@docusaurus/router';+import { useThemeConfig } from '@docusaurus/theme-common';+import useBaseUrl from '@docusaurus/useBaseUrl';+import ThemedImage from '@theme/ThemedImage';+import clsx from 'clsx';+import React from 'react';++import styles from './styles.module.css';++export default function Logo() {+ const {+ navbar: { logo },+ } = useThemeConfig();+ const javascriptLogo = {+ light: useBaseUrl('img/crawlee-javascript-light.svg'),+ dark: useBaseUrl('img/crawlee-javascript-dark.svg'),+ };+ const languageAgnosticLogo = {+ light: useBaseUrl(logo.src),+ dark: useBaseUrl(logo.srcDark || logo.src),+ };+ const location = useLocation();+ const isOnLanguageAgnosticPage = location.pathname === '/' || location.pathname.includes('/blog');+ return (+ <Link+ className={clsx(styles.logoImage, 'sidebarLogo')}+ to={isOnLanguageAgnosticPage ? '/' : '/js'}+ >+ <ThemedImage+ alt={isOnLanguageAgnosticPage ? 'Crawlee' : 'Crawlee JavaScript'}+ sources={isOnLanguageAgnosticPage ? languageAgnosticLogo : javascriptLogo}+ />+ </Link>+ );+}diff --git a/website/src/theme/Logo/styles.module.css b/website/src/theme/Logo/styles.module.cssnew file mode 100644index 000000000..8f6021049--- /dev/null+++ b/website/src/theme/Logo/styles.module.css@@ -0,0 +1,11 @@+.logoImage {+ img {+ display: block !important;+ height: 28px;+ }++ /* do not display the other theme logo when loading */+ img:nth-child(2) {+ display: none !important;+ }+}diff --git a/website/src/theme/Navbar/Content/index.js b/website/src/theme/Navbar/Content/index.jsindex a0f00f9b3..ab415a2a7 100644--- a/website/src/theme/Navbar/Content/index.js+++ b/website/src/theme/Navbar/Content/index.js@@ -1,35 +1,35 @@-import React from 'react';+import Link from '@docusaurus/Link';+import { useLocation } from '@docusaurus/router';import { useThemeConfig } from '@docusaurus/theme-common';import {splitNavbarItems,useNavbarMobileSidebar,} from '@docusaurus/theme-common/internal';-import NavbarItem from '@theme/NavbarItem';-import NavbarColorModeToggle from '@theme/Navbar/ColorModeToggle';-import SearchBar from '@theme/SearchBar';-import NavbarMobileSidebarToggle from '@theme/Navbar/MobileSidebar/Toggle';import NavbarLogo from '@theme/Navbar/Logo';+import NavbarMobileSidebarToggle from '@theme/Navbar/MobileSidebar/Toggle';import NavbarSearch from '@theme/Navbar/Search';+import NavbarItem from '@theme/NavbarItem';+import SearchBar from '@theme/SearchBar';+import clsx from 'clsx';+import React from 'react';+import styles from './styles.module.css';function useNavbarItems() {return useThemeConfig().navbar.items;}-function NavbarItems({ items }) {+function NavbarItems({ items, className }) {return (- <>+ <div className={clsx(styles.navbarItems, className)}>{items.map((item, i) => (- <NavbarItem {...item} key={i}/>+ <NavbarItem {...item} key={i} />))}- </>+ </div>);}-function NavbarContentLayout({- left,- right-}) {+function NavbarContentLayout({ left, right }) {return (<div className="navbar__inner"><div className="navbar__items">{left}</div>@@ -38,29 +38,101 @@ function NavbarContentLayout({);}+const GENERIC_PAGE_ITEMS = [+ {+ to: 'js',+ label: 'JavaScript',+ position: 'left',+ },+ {+ to: 'https://crawlee.dev/python',+ label: 'Python',+ rel: 'dofollow',+ target: '_self',+ position: 'left',+ },+ {+ to: 'blog',+ label: 'Blog',+ position: 'left',+ },+];++const VERSIONS_ITEM = {+ type: 'docsVersionDropdown',+ position: 'left',+ label: 'Versions',+ dropdownItemsAfter: [+ {+ href: 'https://sdk.apify.com/docs/guides/getting-started',+ label: '2.2',+ },+ {+ href: 'https://sdk.apify.com/docs/1.3.1/guides/getting-started',+ label: '1.3',+ },+ ],+ dropdownItemsBefore: [],+};++function getEffectiveNavbarItems(items, location, isOnLanguageAgnosticPage) {+ if (isOnLanguageAgnosticPage) {+ return GENERIC_PAGE_ITEMS;+ }+ if (location.pathname !== '/js' && location.pathname !== '/js/') {+ return [...items, VERSIONS_ITEM];+ }+ return items;+}+export default function NavbarContent() {+ const location = useLocation();+ const isOnLanguageAgnosticPage = location.pathname === '/' || location.pathname.includes('/blog');const mobileSidebar = useNavbarMobileSidebar();const items = useNavbarItems();- const [leftItems, rightItems] = splitNavbarItems(items);+ const effectiveItems = getEffectiveNavbarItems(+ items,+ location,+ isOnLanguageAgnosticPage,+ );+ const [leftItems, rightItems] = splitNavbarItems(effectiveItems);const searchBarItem = items.find((item) => item.type === 'search');return (<NavbarContentLayoutleft={<>- {!mobileSidebar.disabled && <NavbarMobileSidebarToggle/>}- <NavbarLogo/>- <NavbarItems items={leftItems}/>+ <NavbarLogo />+ <NavbarItems+ items={leftItems}+ className={+ isOnLanguageAgnosticPage+ ? styles.navbarItems__leftMargin+ : styles.navbarItems__center+ }+ /></>}right={<>- <NavbarColorModeToggle className={styles.colorModeToggle}/>- <NavbarItems items={rightItems}/>- {!searchBarItem && (- <NavbarSearch>- <SearchBar/>- </NavbarSearch>+ {!isOnLanguageAgnosticPage && (+ <>+ {rightItems?.length > 0 && (+ <NavbarItems items={rightItems} />+ )}+ {!searchBarItem && (+ <NavbarSearch>+ <SearchBar />+ </NavbarSearch>+ )}+ <Link+ className={styles.getStartedButton}+ to="/js/docs/quick-start"+ >+ Get started+ </Link>+ </>)}+ {!mobileSidebar.disabled && <NavbarMobileSidebarToggle />}</>}/>diff --git a/website/src/theme/Navbar/Content/styles.module.css b/website/src/theme/Navbar/Content/styles.module.cssindex 4c9471e10..220e3540b 100644--- a/website/src/theme/Navbar/Content/styles.module.css+++ b/website/src/theme/Navbar/Content/styles.module.css@@ -1,8 +1,31 @@-/*-Hide color mode toggle in small viewports- */-@media (max-width: 996px) {- .colorModeToggle {- display: none;- }+.navbarItems {+ display: flex;+ align-items: center;+ gap: 40px;+}++.navbarItems__center {+ margin-inline: auto;+ gap: 16px;+}++.navbarItems__leftMargin {+ margin-left: 40px;+}++.getStartedButton {+ color: var(--color-text-on-primary);+ background: var(--color-black-action);+ border-radius: 8px;+ font-size: 16px;+ font-weight: 500;+ line-height: 24px;+ padding: 8px 16px !important;+ border: none;+ transition: background-color 0.2s;++ &:hover {+ color: var(--color-text-on-primary);+ background-color: var(--color-primary-action-hover);+ }}diff --git a/website/src/theme/Navbar/Logo/index.js b/website/src/theme/Navbar/Logo/index.jsnew file mode 100644index 000000000..c6bfe2dad--- /dev/null+++ b/website/src/theme/Navbar/Logo/index.js@@ -0,0 +1,52 @@+import Link from '@docusaurus/Link';+import { useLocation } from '@docusaurus/router';+import { useThemeConfig } from '@docusaurus/theme-common';+import useBaseUrl from '@docusaurus/useBaseUrl';+import Logo from '@theme/Logo';+import ThemedImage from '@theme/ThemedImage';+import React from 'react';++import styles from './index.module.css';++export default function LogoWrapper(props) {+ const ArrowsIcon = require('../../../../static/img/menu-arrows.svg').default;+ const CheckIcon = require('../../../../static/img/check.svg').default;+ const { navbar: { logo } } = useThemeConfig();+ const javascriptLogo = {+ light: useBaseUrl('img/crawlee-javascript-light.svg'),+ dark: useBaseUrl('img/crawlee-javascript-dark.svg'),+ };+ const pythonLogo = {+ light: useBaseUrl('img/crawlee-python-light.svg'),+ dark: useBaseUrl('img/crawlee-python-dark.svg'),+ };+ const languageAgnosticLogo = {+ light: useBaseUrl(logo.src),+ dark: useBaseUrl(logo.srcDark || logo.src),+ };+ const location = useLocation();+ const isOnLanguageAgnosticPage = location.pathname === '/' || location.pathname.includes('/blog');+ return (+ <div className={styles.navbarLogo}>+ <div className={styles.logoWithArrows}>+ <Logo />+ <ArrowsIcon />+ </div>+ <div className={styles.menuWrapper}>+ <div className={styles.menu}>+ <Link className={styles.menuItem} to="/js">+ <ThemedImage sources={javascriptLogo} alt="Crawlee JavaScript" />+ {!isOnLanguageAgnosticPage && <CheckIcon />}+ </Link>+ <Link className={styles.menuItem} href="https://crawlee.dev/python" target="_self" rel="dofollow">+ <ThemedImage sources={pythonLogo} alt="Crawlee Python" />+ </Link>+ <Link className={styles.menuItem} to="/">+ <ThemedImage sources={languageAgnosticLogo} alt="Crawlee" />+ {isOnLanguageAgnosticPage && <CheckIcon />}+ </Link>+ </div>+ </div>+ </div >+ );+}diff --git a/website/src/theme/Navbar/Logo/index.module.css b/website/src/theme/Navbar/Logo/index.module.cssnew file mode 100644index 000000000..82e2d8439--- /dev/null+++ b/website/src/theme/Navbar/Logo/index.module.css@@ -0,0 +1,74 @@+.navbarLogo {+ position: relative;+ cursor: pointer;+}++.logoWithArrows {+ display: flex;+ align-items: center;++ svg {+ margin: 0 2px;+ g {+ stroke: var(--color-icon);+ }+ }+}++.menuWrapper {+ position: absolute;+ left: 0;+ top: 100%;++ z-index: 100;+ padding-top: 6px;+}++.menu {+ width: 230px;+ border-radius: 8px;+ border: 1px solid var(--color-border);+ box-shadow:+ 0px 4px 8px 0px rgba(36, 39, 54, 0.12),+ 0px 2px 4px 0px rgba(36, 39, 54, 0.08),+ 0px 0px 1px 0px rgba(36, 39, 54, 0.24);++ background: var(--color-card-background);+ padding: 8px 0;+ overflow: hidden;+ transition: all 0.3s;++ flex-direction: column;+ align-items: flex-start;++ padding: 8px;++ display: none;++ img {+ height: 24px;+ width: auto;+ display: block !important;+ }+}++.navbarLogo:hover {+ .menu {+ display: flex;+ }+}++.menuItem {+ padding: 8px;+ width: 100%;+ border-radius: 12px;+ display: flex;+ justify-content: space-between;+ align-items: center;+ path {+ fill: var(--color-icon);+ }+ &:hover {+ background: var(--color-hover);+ }+}diff --git a/website/src/theme/Navbar/MobileSidebar/Header/index.js b/website/src/theme/Navbar/MobileSidebar/Header/index.jsnew file mode 100644index 000000000..d6de3ab8e--- /dev/null+++ b/website/src/theme/Navbar/MobileSidebar/Header/index.js@@ -0,0 +1,61 @@+import Link from '@docusaurus/Link';+import { useLocation } from '@docusaurus/router';+import { useNavbarMobileSidebar } from '@docusaurus/theme-common/internal';+import { translate } from '@docusaurus/Translate';+import IconClose from '@theme/Icon/Close';+import NavbarLogo from '@theme/Navbar/Logo';+import SearchBar from '@theme/SearchBar';+import clsx from 'clsx';+import React from 'react';++import styles from './index.module.css';++function CloseButton() {+ const mobileSidebar = useNavbarMobileSidebar();+ return (+ <button+ type="button"+ aria-label={translate({+ id: 'theme.docs.sidebar.closeSidebarButtonAriaLabel',+ message: 'Close navigation bar',+ description: 'The ARIA label for close button of mobile sidebar',+ })}+ className="clean-btn navbar-sidebar__close"+ onClick={() => mobileSidebar.toggle()}>+ <IconClose color="var(--ifm-color-emphasis-600)" />+ </button>+ );+}+export default function NavbarMobileSidebarHeader() {+ const location = useLocation();+ const isOnLanguageAgnosticPage = location.pathname === '/' || location.pathname.includes('/blog');+ const { toggle, shown } = useNavbarMobileSidebar();+ const closeSidebar = () => shown && toggle();++ return (+ <div className="navbar-sidebar__brand">+ <div className={styles.navbarHeader}>+ <NavbarLogo />++ {!isOnLanguageAgnosticPage && <div className={clsx(styles.navbarButtonsWrapper, styles.navbarButtonsWrapperDesktop)} >+ <div onClick={closeSidebar} >+ <SearchBar />+ </div>+ <Link className={styles.getStartedButton} to="/js/docs/quick-start" onClick={closeSidebar} >+ Get started+ </Link>+ </div>}+ <CloseButton />+ </div>+ {!isOnLanguageAgnosticPage && <div className={clsx(styles.navbarButtonsWrapper, styles.navbarButtonsWrapperMobile)} >+ <Link className={styles.getStartedButton} to="/js/docs/quick-start" onClick={closeSidebar}>+ Get started+ </Link>+ <div onClick={closeSidebar} >+ <SearchBar />+ </div>+ </div>}++ </div>+ );+}diff --git a/website/src/theme/Navbar/MobileSidebar/Header/index.module.css b/website/src/theme/Navbar/MobileSidebar/Header/index.module.cssnew file mode 100644index 000000000..54a247d46--- /dev/null+++ b/website/src/theme/Navbar/MobileSidebar/Header/index.module.css@@ -0,0 +1,57 @@+.getStartedButton {+ color: var(--color-text-on-primary);+ background: var(--color-black-action);+ border-radius: 8px;+ font-size: 16px;+ font-weight: 500;+ line-height: 24px;+ padding: 8px 16px !important;+ border: none;+ &:hover {+ color: var(--color-text-on-primary);+ }+ text-align: center;+}++.navbarHeader {+ display: flex;+ width: 100%;+ align-items: center;+ justify-content: space-between;+ padding: 16px;++ @media (min-width: 768px) {+ padding: 20px 40px;+ }+ @media (min-width: 1024px) {+ padding: 20px 64px;+ }+}++.navbarButtonsWrapper {+ display: flex;+ gap: 16px;+ margin-left: auto;+}++.navbarButtonsWrapperDesktop {+ display: flex;+ @media (max-width: 767px) {+ display: none;+ }+}+.navbarButtonsWrapperMobile {+ border-top: 1px solid var(--color-separator);+ display: none;+ @media (max-width: 767px) {+ display: flex;+ }+ width: 100%;+ margin: 0;+ flex-direction: column;+ gap: 16px;+ button {+ width: 100%;+ }+ padding: 16px 24px;+}diff --git a/website/src/theme/Navbar/MobileSidebar/Layout/index.js b/website/src/theme/Navbar/MobileSidebar/Layout/index.jsnew file mode 100644index 000000000..91b74ea5c--- /dev/null+++ b/website/src/theme/Navbar/MobileSidebar/Layout/index.js@@ -0,0 +1,23 @@+import { useNavbarSecondaryMenu } from '@docusaurus/theme-common/internal';+import clsx from 'clsx';+import React from 'react';++export default function NavbarMobileSidebarLayout({+ header,+ primaryMenu,+ secondaryMenu,+}) {+ const { shown: secondaryMenuShown } = useNavbarSecondaryMenu();+ return (+ <div className="navbar-sidebar">+ {header}+ <div+ className={clsx('navbar-sidebar__items', {+ 'navbar-sidebar__items--show-secondary': secondaryMenuShown,+ })}>+ <div className="navbar-sidebar__item menu menu-primary">{primaryMenu}</div>+ <div className="navbar-sidebar__item menu menu-secondary">{secondaryMenu}</div>+ </div>+ </div>+ );+}diff --git a/website/src/theme/Navbar/MobileSidebar/PrimaryMenu/index.js b/website/src/theme/Navbar/MobileSidebar/PrimaryMenu/index.jsnew file mode 100644index 000000000..2bc774049--- /dev/null+++ b/website/src/theme/Navbar/MobileSidebar/PrimaryMenu/index.js@@ -0,0 +1,47 @@+import { useLocation } from '@docusaurus/router';+import { useThemeConfig } from '@docusaurus/theme-common';+import { useNavbarMobileSidebar } from '@docusaurus/theme-common/internal';+import NavbarItem from '@theme/NavbarItem';+import React from 'react';++const GENERIC_PAGE_ITEMS = [{+ to: 'js',+ label: 'JavaScript',+ position: 'left',+}, {+ to: 'https://crawlee.dev/python',+ label: 'Python',+ rel: 'dofollow',+ target: '_self',+ position: 'left',+},+{+ to: 'blog',+ label: 'Blog',+ position: 'left',+}];++function useNavbarItems() {+ const location = useLocation();+ const defaultItems = useThemeConfig().navbar.items;+ const isOnLanguageAgnosticPage = location.pathname === '/' || location.pathname.includes('/blog');+ return isOnLanguageAgnosticPage ? GENERIC_PAGE_ITEMS : defaultItems;+}+// The primary menu displays the navbar items+export default function NavbarMobilePrimaryMenu() {+ const mobileSidebar = useNavbarMobileSidebar();+ const items = useNavbarItems();++ return (+ <ul className="menu__list">+ {items.map((item, i) => (+ <NavbarItem+ mobile+ {...item}+ onClick={() => mobileSidebar.toggle()}+ key={i}+ />+ ))}+ </ul>+ );+}diff --git a/website/src/theme/Navbar/MobileSidebar/index.js b/website/src/theme/Navbar/MobileSidebar/index.jsnew file mode 100644index 000000000..ad92c6d53--- /dev/null+++ b/website/src/theme/Navbar/MobileSidebar/index.js@@ -0,0 +1,30 @@+import {+ useLockBodyScroll,+ useNavbarMobileSidebar,+ useWindowSize,+} from '@docusaurus/theme-common/internal';+import NavbarMobileSidebarHeader from '@theme/Navbar/MobileSidebar/Header';+import NavbarMobileSidebarLayout from '@theme/Navbar/MobileSidebar/Layout';+import NavbarMobileSidebarPrimaryMenu from '@theme/Navbar/MobileSidebar/PrimaryMenu';+import NavbarMobileSidebarSecondaryMenu from '@theme/Navbar/MobileSidebar/SecondaryMenu';+import React from 'react';++export default function NavbarMobileSidebar() {+ const mobileSidebar = useNavbarMobileSidebar();+ const windowSize = useWindowSize({+ desktopBreakpoint: 1200,+ });++ useLockBodyScroll(mobileSidebar.shown);+ const shouldRender = !mobileSidebar.disabled && windowSize === 'mobile';+ if (!shouldRender) {+ return null;+ }+ return (+ <NavbarMobileSidebarLayout+ header={<NavbarMobileSidebarHeader />}+ primaryMenu={<NavbarMobileSidebarPrimaryMenu />}+ secondaryMenu={<NavbarMobileSidebarSecondaryMenu />}+ />+ );+}diff --git a/website/src/theme/NavbarItem/ComponentTypes.js b/website/src/theme/NavbarItem/ComponentTypes.jsindex 04a5d0362..d0a4e9d36 100644--- a/website/src/theme/NavbarItem/ComponentTypes.js+++ b/website/src/theme/NavbarItem/ComponentTypes.js@@ -55,7 +55,7 @@ function ApiNavbarItem(ctx) {exact{...ctx}label={ctx.label}- to={`api/${ctx.to}`}+ to={`js/api/${ctx.to}`}/>);}@@ -65,7 +65,7 @@ function ApiNavbarItem(ctx) {exact{...ctx}label={ctx.label}- to={`api/${version.version === 'current' ? 'next' : version.version}/${ctx.to}`}+ to={`js/api/${version.version === 'current' ? 'next' : version.version}/${ctx.to}`}/>);}diff --git a/website/static/img/arrow_right.svg b/website/static/img/arrow_right.svgnew file mode 100644index 000000000..37e30d2e2--- /dev/null+++ b/website/static/img/arrow_right.svg@@ -0,0 +1,8 @@+<svg width="20" height="20" viewBox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg">+ <path d="M4.16687 10H15.8335" stroke="#555D76" stroke-width="1.66667" stroke-linecap="round"+ stroke-linejoin="round" />+ <path d="M10.8331 15L15.8331 10" stroke="#555D76" stroke-width="1.66667" stroke-linecap="round"+ stroke-linejoin="round" />+ <path d="M10.8331 5L15.8331 10" stroke="#555D76" stroke-width="1.66667" stroke-linecap="round"+ stroke-linejoin="round" />+</svg>\ No newline at end of filediff --git a/website/static/img/auto-scaling-dark.webp b/website/static/img/auto-scaling-dark.webpnew file mode 100644index 000000000..942f7b1ffBinary files /dev/null and b/website/static/img/auto-scaling-dark.webp differdiff --git a/website/static/img/auto-scaling-light.webp b/website/static/img/auto-scaling-light.webpnew file mode 100644index 000000000..a93153df0Binary files /dev/null and b/website/static/img/auto-scaling-light.webp differdiff --git a/website/static/img/check.svg b/website/static/img/check.svgnew file mode 100644index 000000000..27e530e18--- /dev/null+++ b/website/static/img/check.svg@@ -0,0 +1,5 @@+<svg width="17" height="16" viewBox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg">+<g id="Check_16">+<path id="shape" d="M14.7419 4.22032C14.8824 4.36094 14.9612 4.55157 14.9612 4.75032C14.9612 4.94907 14.8824 5.13969 14.7419 5.28032L7.4919 12.5303C7.35128 12.6708 7.16065 12.7497 6.9619 12.7497C6.76315 12.7497 6.57253 12.6708 6.4319 12.5303L3.1819 9.28032C3.0495 9.13805 2.97739 8.95 2.98075 8.75569C2.98411 8.56137 3.06266 8.37592 3.1999 8.23832C3.33751 8.10107 3.52296 8.02252 3.71727 8.01916C3.91159 8.0158 4.09964 8.08791 4.2419 8.22032L6.9619 10.9403L13.6819 4.22032C13.8225 4.07987 14.0132 4.00098 14.2119 4.00098C14.4107 4.00098 14.6013 4.07987 14.7419 4.22032Z" fill="#555D76"/>+</g>+</svg>diff --git a/website/static/img/community-dark-icon.svg b/website/static/img/community-dark-icon.svgnew file mode 100644index 000000000..d7ab5225c--- /dev/null+++ b/website/static/img/community-dark-icon.svg@@ -0,0 +1,7 @@+<svg width="52" height="52" viewBox="0 0 52 52" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="18.7153" cy="20.9995" r="11.665" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="32.7153" cy="20.9995" r="11.665" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2 2"/>+<rect x="9.42244" y="27.9683" width="23" height="23" transform="rotate(-45 9.42244 27.9683)" fill="#1A1B21" stroke="#B2B8CC"/>+<path d="M28.5005 25.4995L23.5005 30.4995" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M24.0005 25.4995H28.5005V29.9995" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/community-light-icon.svg b/website/static/img/community-light-icon.svgnew file mode 100644index 000000000..5159a6100--- /dev/null+++ b/website/static/img/community-light-icon.svg@@ -0,0 +1,7 @@+<svg width="52" height="52" viewBox="0 0 52 52" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="18.7155" cy="20.9995" r="11.665" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="32.7155" cy="20.9995" r="11.665" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2 2"/>+<rect x="9.42256" y="27.9683" width="23" height="23" transform="rotate(-45 9.42256 27.9683)" fill="white" stroke="#555D76"/>+<path d="M28.5006 25.4995L23.5006 30.4995" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M24.0006 25.4995H28.5006V29.9995" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/crawlee-dark.svg b/website/static/img/crawlee-dark.svgindex ffce091ca..f5ac71eca 100644--- a/website/static/img/crawlee-dark.svg+++ b/website/static/img/crawlee-dark.svg@@ -1,66 +1,66 @@-<svg width="138" height="42" fill="none" xmlns="http://www.w3.org/2000/svg">- <g clip-path="url(#a)" stroke-linecap="round" stroke-linejoin="round">- <path d="M39.843 34.81h-29.4c-.42 0-.683-.474-.473-.855l7.35-13.24 7.351-13.238a.537.537 0 0 1 .947 0l4.728 8.517a6.521 6.521 0 0 0 1.57 12.848c1.765 0 3.369-.7 4.542-1.843l3.861 6.956c.21.378-.053.854-.473.854h-.003Z" fill="url(#b)" stroke="url(#c)" stroke-width="1.039"/>- <path d="M37.855 25.017a6.519 6.519 0 0 1-5.938 3.825 6.518 6.518 0 0 1-6.52-6.52 6.518 6.518 0 0 1 9.343-5.878" stroke="url(#d)" stroke-width="2"/>- </g>- <path d="M52.868 27.614c3.223 0 4.891-1.897 5.188-4.343h-2.742c-.206 1.12-.846 2.035-2.332 2.035-1.691 0-2.72-1.257-2.72-3.406s1.097-3.406 2.743-3.406c1.417 0 2.103.937 2.309 2.103h2.742c-.274-2.514-1.92-4.411-5.12-4.411-3.222 0-5.508 2.377-5.508 5.76 0 3.314 2.057 5.668 5.44 5.668ZM65.9 16.186c-1.44 0-2.354.8-2.902 1.92l-.046-1.806h-2.674v11.2h2.788v-5.989c.183-1.691 1.349-2.628 2.72-2.628.434 0 .617.046.777.114l.023-2.743a4.258 4.258 0 0 0-.686-.068Zm12.952 8.96a1.483 1.483 0 0 1-.572.114c-.343 0-.571-.251-.571-.686v-3.931c0-2.789-1.715-4.457-4.892-4.457-2.857 0-4.8 1.394-4.983 3.908h2.56c.115-1.097.938-1.737 2.263-1.737 1.555 0 2.332.823 2.332 2.126v.571h-1.737c-3.063 0-5.692.869-5.692 3.52 0 1.806 1.303 3.04 3.794 3.04 2.4 0 3.383-1.143 3.749-1.828.32 1.211 1.211 1.828 2.423 1.828.571 0 1.074-.091 1.508-.205l-.182-2.263Zm-6.766.388c-1.234 0-1.806-.457-1.806-1.257 0-1.051 1.006-1.6 2.834-1.6h1.875v.732c0 .96-1.075 2.125-2.903 2.125ZM93.525 16.3l-1.692 7.154c-.16.595-.251 1.006-.297 1.349h-.069c-.068-.343-.182-.8-.32-1.349l-1.85-7.154h-2.584l-1.828 7.154c-.138.549-.275 1.006-.32 1.349h-.069c-.046-.343-.183-.8-.297-1.349L82.485 16.3h-2.949l3.269 11.2h3.2l1.782-6.514c.07-.297.16-.846.183-1.075h.069c.023.229.137.778.206 1.075l1.782 6.514h3.2l3.246-11.2h-2.948Zm8.584 8.846c-.252.091-.366.114-.549.114-.343 0-.594-.251-.594-.686V11.271H98.2v13.532c0 1.783.892 2.811 2.56 2.811.617 0 1.097-.091 1.532-.205l-.183-2.263Zm12.335-3.177c0-3.498-1.92-5.783-5.235-5.783-3.2 0-5.531 2.354-5.531 5.737 0 3.314 2.08 5.691 5.508 5.691 3.269 0 4.823-1.805 5.212-3.771h-2.629c-.183.686-.8 1.554-2.468 1.554-1.6 0-2.675-1.028-2.88-2.857h8.023v-.571Zm-5.212-3.635c1.394 0 2.332.777 2.56 2.492h-5.326c.298-1.646 1.372-2.492 2.766-2.492Zm17.689 3.635c0-3.498-1.92-5.783-5.234-5.783-3.2 0-5.531 2.354-5.531 5.737 0 3.314 2.08 5.691 5.508 5.691 3.269 0 4.823-1.805 5.212-3.771h-2.629c-.183.686-.8 1.554-2.469 1.554-1.6 0-2.674-1.028-2.88-2.857h8.023v-.571Zm-5.211-3.635c1.394 0 2.331.777 2.56 2.492h-5.326c.297-1.646 1.372-2.492 2.766-2.492Z" fill="#F2F3FB"/>- <defs>- <linearGradient id="b" x1="40.393" y1="7.193" x2="12.912" y2="37.541" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <linearGradient id="c" x1="40.393" y1="7.193" x2="12.912" y2="37.541" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <linearGradient id="d" x1="37.855" y1="22.323" x2="25.398" y2="22.323" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <clipPath id="a">- <path fill="#fff" d="M9.418 6.714h31.45v28.571H9.418z"/>- </clipPath>- </defs>+<svg width="106" height="28" viewBox="0 0 106 28" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_2100)">+<path d="M27.0255 26H1.47554C1.11002 26 0.88156 25.5888 1.06432 25.2575L7.45252 13.7519L13.8407 2.2463C14.0235 1.9179 14.4804 1.9179 14.6632 2.2463L18.7725 9.64827C16.3023 10.2594 14.4718 12.4897 14.4718 15.1483C14.4718 18.2782 17.0077 20.8141 20.1375 20.8141C21.671 20.8141 23.0646 20.2058 24.0841 19.212L27.4396 25.2575C27.6223 25.5859 27.3939 26 27.0283 26H27.0255Z" fill="url(#paint0_linear_1184_2100)" stroke="url(#paint1_linear_1184_2100)" stroke-width="1.01449" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.826 17.0074C24.935 18.9692 22.9588 20.3314 20.6657 20.3314C17.5359 20.3314 15 17.7955 15 14.6657C15 11.5359 17.5359 9 20.6657 9C21.5453 9 22.3763 9.1999 23.1187 9.55686" stroke="url(#paint2_linear_1184_2100)" stroke-width="1.95228" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M39.5 20.1C36.54 20.1 34.74 18.04 34.74 15.14C34.74 12.18 36.74 10.1 39.56 10.1C42.36 10.1 43.8 11.76 44.04 13.96H41.64C41.46 12.94 40.86 12.12 39.62 12.12C38.18 12.12 37.22 13.22 37.22 15.1C37.22 16.98 38.12 18.08 39.6 18.08C40.9 18.08 41.46 17.28 41.64 16.3H44.04C43.78 18.44 42.32 20.1 39.5 20.1ZM50.9036 10.1C51.1436 10.1 51.3836 10.14 51.5036 10.16L51.4836 12.56C51.3436 12.5 51.1836 12.46 50.8036 12.46C49.6036 12.46 48.5836 13.28 48.4236 14.76V20H45.9836V10.2H48.3236L48.3636 11.78C48.8436 10.8 49.6436 10.1 50.9036 10.1ZM62.2358 17.94L62.3958 19.92C62.0158 20.02 61.5758 20.1 61.0758 20.1C60.0158 20.1 59.2358 19.56 58.9558 18.5C58.6358 19.1 57.7758 20.1 55.6758 20.1C53.4958 20.1 52.3558 19.02 52.3558 17.44C52.3558 15.12 54.6558 14.36 57.3358 14.36H58.8558V13.86C58.8558 12.72 58.1758 12 56.8158 12C55.6558 12 54.9358 12.56 54.8358 13.52H52.5958C52.7558 11.32 54.4558 10.1 56.9558 10.1C59.7358 10.1 61.2358 11.56 61.2358 14V17.44C61.2358 17.82 61.4358 18.04 61.7358 18.04C61.8958 18.04 62.0158 18.02 62.2358 17.94ZM56.3158 18.28C57.9158 18.28 58.8558 17.26 58.8558 16.42V15.78H57.2158C55.6158 15.78 54.7358 16.26 54.7358 17.18C54.7358 17.88 55.2358 18.28 56.3158 18.28ZM75.0747 10.2H77.6547L74.8147 20H72.0147L70.4547 14.3C70.3947 14.04 70.2947 13.56 70.2747 13.36H70.2147C70.1947 13.56 70.1147 14.04 70.0547 14.3L68.4947 20H65.6947L62.8347 10.2H65.4147L66.9147 16.46C67.0147 16.94 67.1347 17.34 67.1747 17.64H67.2347C67.2747 17.34 67.3947 16.94 67.5147 16.46L69.1147 10.2H71.3747L72.9947 16.46C73.1147 16.94 73.2147 17.34 73.2747 17.64H73.3347C73.3747 17.34 73.4547 16.98 73.5947 16.46L75.0747 10.2ZM82.5858 17.94L82.7458 19.92C82.3658 20.02 81.9458 20.1 81.4058 20.1C79.9458 20.1 79.1658 19.2 79.1658 17.64V5.8H81.5858V17.44C81.5858 17.82 81.8058 18.04 82.1058 18.04C82.2658 18.04 82.3658 18.02 82.5858 17.94ZM93.3788 15.16V15.66H86.3588C86.5388 17.26 87.4788 18.16 88.8788 18.16C90.3388 18.16 90.8788 17.4 91.0388 16.8H93.3388C92.9988 18.52 91.6388 20.1 88.7788 20.1C85.7788 20.1 83.9588 18.02 83.9588 15.12C83.9588 12.16 85.9988 10.1 88.7988 10.1C91.6988 10.1 93.3788 12.1 93.3788 15.16ZM88.8188 11.98C87.5988 11.98 86.6588 12.72 86.3988 14.16H91.0588C90.8588 12.66 90.0388 11.98 88.8188 11.98ZM104.297 15.16V15.66H97.2767C97.4567 17.26 98.3967 18.16 99.7967 18.16C101.257 18.16 101.797 17.4 101.957 16.8H104.257C103.917 18.52 102.557 20.1 99.6967 20.1C96.6967 20.1 94.8767 18.02 94.8767 15.12C94.8767 12.16 96.9167 10.1 99.7167 10.1C102.617 10.1 104.297 12.1 104.297 15.16ZM99.7367 11.98C98.5167 11.98 97.5767 12.72 97.3167 14.16H101.977C101.777 12.66 100.957 11.98 99.7367 11.98Z" fill="#F3F4FA"/>+<defs>+<linearGradient id="paint0_linear_1184_2100" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_2100" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_2100" x1="25.826" y1="9" x2="14.5063" y2="19.8147" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_2100">+<rect width="28" height="28" fill="white"/>+</clipPath>+</defs></svg>diff --git a/website/static/img/crawlee-javascript-dark.svg b/website/static/img/crawlee-javascript-dark.svgnew file mode 100644index 000000000..c85c428a9--- /dev/null+++ b/website/static/img/crawlee-javascript-dark.svg@@ -0,0 +1,67 @@+<svg width="167" height="24" viewBox="0 0 167 24" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_806)">+<path d="M23.0208 22.222H1.25688C0.94552 22.222 0.750917 21.8717 0.906599 21.5895L6.34819 11.7888L11.7898 1.98813C11.9455 1.70838 12.3347 1.70838 12.4903 1.98813L15.9908 8.29326C13.8866 8.81383 12.3274 10.7136 12.3274 12.9783C12.3274 15.6444 14.4875 17.8045 17.1535 17.8045C18.4598 17.8045 19.6469 17.2864 20.5153 16.4398L23.3735 21.5895C23.5292 21.8693 23.3346 22.222 23.0232 22.222H23.0208Z" fill="url(#paint0_linear_1184_806)" stroke="url(#paint1_linear_1184_806)" stroke-width="0.864159" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M21.9988 14.562C21.2398 16.2332 19.5565 17.3935 17.6032 17.3935C14.9371 17.3935 12.777 15.2334 12.777 12.5674C12.777 9.9013 14.9371 7.74121 17.6032 7.74121C18.3524 7.74121 19.0603 7.91149 19.6927 8.21556" stroke="url(#paint2_linear_1184_806)" stroke-width="1.66299" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M33.6469 17.0852C31.1255 17.0852 29.5922 15.3304 29.5922 12.8602C29.5922 10.3388 31.2958 8.56699 33.698 8.56699C36.0831 8.56699 37.3097 9.98101 37.5141 11.855H35.4698C35.3164 10.9862 34.8053 10.2877 33.7491 10.2877C32.5225 10.2877 31.7047 11.2247 31.7047 12.8261C31.7047 14.4275 32.4713 15.3645 33.732 15.3645C34.8394 15.3645 35.3164 14.6831 35.4698 13.8483H37.5141C37.2926 15.6712 36.049 17.0852 33.6469 17.0852ZM43.3607 8.56699C43.5651 8.56699 43.7695 8.60106 43.8717 8.6181L43.8547 10.6625C43.7355 10.6114 43.5992 10.5773 43.2755 10.5773C42.2533 10.5773 41.3844 11.2758 41.2481 12.5365V17H39.1697V8.65217H41.163L41.197 9.99805C41.6059 9.16327 42.2874 8.56699 43.3607 8.56699ZM53.0136 15.2453L53.1499 16.9319C52.8262 17.017 52.4514 17.0852 52.0255 17.0852C51.1226 17.0852 50.4582 16.6252 50.2197 15.7223C49.9471 16.2334 49.2145 17.0852 47.4257 17.0852C45.5687 17.0852 44.5977 16.1652 44.5977 14.8193C44.5977 12.8431 46.5568 12.1957 48.8397 12.1957H50.1345V11.7698C50.1345 10.7988 49.5552 10.1854 48.3968 10.1854C47.4087 10.1854 46.7953 10.6625 46.7102 11.4802H44.8021C44.9384 9.60621 46.3865 8.56699 48.516 8.56699C50.8841 8.56699 52.1618 9.81065 52.1618 11.8891V14.8193C52.1618 15.143 52.3322 15.3304 52.5877 15.3304C52.724 15.3304 52.8262 15.3134 53.0136 15.2453ZM47.9709 15.5349C49.3338 15.5349 50.1345 14.666 50.1345 13.9505V13.4053H48.7375C47.3746 13.4053 46.625 13.8142 46.625 14.5979C46.625 15.1941 47.0509 15.5349 47.9709 15.5349ZM63.9501 8.65217H66.1477L63.7286 17H61.3435L60.0146 12.1446C59.9635 11.9232 59.8784 11.5143 59.8613 11.3439H59.8102C59.7932 11.5143 59.725 11.9232 59.6739 12.1446L58.3451 17H55.96L53.5238 8.65217H55.7215L56.9992 13.9846C57.0844 14.3934 57.1866 14.7342 57.2207 14.9897H57.2718C57.3059 14.7342 57.4081 14.3934 57.5103 13.9846L58.8732 8.65217H60.7983L62.1783 13.9846C62.2805 14.3934 62.3657 14.7342 62.4168 14.9897H62.4679C62.502 14.7342 62.5701 14.4275 62.6894 13.9846L63.9501 8.65217ZM70.3481 15.2453L70.4844 16.9319C70.1607 17.017 69.803 17.0852 69.343 17.0852C68.0993 17.0852 67.4349 16.3185 67.4349 14.9897V4.90417H69.4963V14.8193C69.4963 15.143 69.6837 15.3304 69.9393 15.3304C70.0756 15.3304 70.1607 15.3134 70.3481 15.2453ZM79.5418 12.8772V13.3031H73.562C73.7154 14.666 74.5161 15.4327 75.7086 15.4327C76.9523 15.4327 77.4123 14.7853 77.5485 14.2742H79.5077C79.2181 15.7393 78.0596 17.0852 75.6234 17.0852C73.068 17.0852 71.5177 15.3134 71.5177 12.8431C71.5177 10.3217 73.2554 8.56699 75.6405 8.56699C78.1107 8.56699 79.5418 10.2706 79.5418 12.8772ZM75.6575 10.1684C74.6183 10.1684 73.8176 10.7988 73.5961 12.0254H77.5656C77.3952 10.7476 76.6967 10.1684 75.6575 10.1684ZM88.8419 12.8772V13.3031H82.8622C83.0155 14.666 83.8162 15.4327 85.0087 15.4327C86.2524 15.4327 86.7124 14.7853 86.8487 14.2742H88.8079C88.5182 15.7393 87.3598 17.0852 84.9236 17.0852C82.3681 17.0852 80.8178 15.3134 80.8178 12.8431C80.8178 10.3217 82.5555 8.56699 84.9406 8.56699C87.4109 8.56699 88.8419 10.2706 88.8419 12.8772ZM84.9576 10.1684C83.9184 10.1684 83.1177 10.7988 82.8962 12.0254H86.8657C86.6953 10.7476 85.9969 10.1684 84.9576 10.1684ZM103.155 14.1106C103.168 15.4872 103.918 16.0051 104.749 16.0051C105.69 16.0051 106.344 15.3781 106.344 14.1106V7.65043H107.543V14.0289C107.543 15.8279 106.589 17.0545 104.763 17.0545C102.923 17.0545 101.955 15.9506 101.955 14.1106H103.155ZM115.663 15.9097L115.759 16.9591C115.527 17.0273 115.282 17.0681 114.996 17.0681C114.301 17.0681 113.837 16.632 113.755 15.9778C113.51 16.3594 112.897 17.0681 111.411 17.0681C109.898 17.0681 109.149 16.3185 109.149 15.2555C109.149 13.8108 110.484 13.2111 112.338 13.2111H113.742V12.775C113.742 11.8482 113.169 11.2758 112.093 11.2758C111.139 11.2758 110.553 11.7255 110.471 12.5297H109.299C109.408 11.085 110.539 10.2536 112.174 10.2536C113.933 10.2536 114.941 11.194 114.941 12.8022V15.5281C114.941 15.8143 115.105 15.9778 115.323 15.9778C115.445 15.9778 115.527 15.9642 115.663 15.9097ZM111.657 16.1005C113.006 16.1005 113.742 15.1601 113.742 14.5468V14.0834H112.27C111.016 14.0834 110.375 14.465 110.375 15.201C110.375 15.7598 110.757 16.1005 111.657 16.1005ZM121.256 10.3217H122.482L119.92 17H118.612L116.009 10.3217H117.358L119.075 15.0647C119.171 15.31 119.225 15.5008 119.293 15.7734H119.32C119.375 15.5008 119.429 15.31 119.511 15.0647L121.256 10.3217ZM129.612 15.9097L129.707 16.9591C129.476 17.0273 129.23 17.0681 128.944 17.0681C128.249 17.0681 127.786 16.632 127.704 15.9778C127.459 16.3594 126.845 17.0681 125.36 17.0681C123.847 17.0681 123.097 16.3185 123.097 15.2555C123.097 13.8108 124.433 13.2111 126.287 13.2111H127.69V12.775C127.69 11.8482 127.118 11.2758 126.041 11.2758C125.087 11.2758 124.501 11.7255 124.419 12.5297H123.247C123.356 11.085 124.487 10.2536 126.123 10.2536C127.881 10.2536 128.89 11.194 128.89 12.8022V15.5281C128.89 15.8143 129.053 15.9778 129.271 15.9778C129.394 15.9778 129.476 15.9642 129.612 15.9097ZM125.605 16.1005C126.954 16.1005 127.69 15.1601 127.69 14.5468V14.0834H126.218C124.964 14.0834 124.324 14.465 124.324 15.201C124.324 15.7598 124.706 16.1005 125.605 16.1005ZM132.023 10.0628C132.023 12.5842 137.529 10.7034 137.529 14.5604C137.529 16.0732 136.139 17.0818 134.081 17.0818C131.887 17.0818 130.565 16.0869 130.565 14.3151H131.75C131.75 15.3781 132.595 16.0187 134.176 16.0187C135.512 16.0187 136.33 15.4872 136.316 14.5604C136.316 11.9163 130.81 13.838 130.81 10.09C130.81 8.54996 132.146 7.56866 134.149 7.56866C136.153 7.56866 137.42 8.56359 137.406 10.2672H136.221C136.221 9.13601 135.389 8.63173 134.108 8.63173C132.664 8.63173 132.023 9.14964 132.023 10.0628ZM141.871 17.0681C139.841 17.0681 138.75 15.6507 138.75 13.6881C138.75 11.6846 140.045 10.2536 141.94 10.2536C143.684 10.2536 144.706 11.2349 144.884 12.7613H143.602C143.466 11.93 142.962 11.2758 141.953 11.2758C140.754 11.2758 139.977 12.2162 139.977 13.6609C139.977 15.0919 140.686 16.046 141.926 16.046C142.962 16.046 143.466 15.3918 143.602 14.6013H144.884C144.679 16.0869 143.643 17.0681 141.871 17.0681ZM149.303 10.2536C149.48 10.2536 149.603 10.2809 149.685 10.2945L149.671 11.4802C149.617 11.453 149.494 11.4257 149.221 11.4257C148.376 11.4257 147.586 12.1344 147.586 13.4837V17H146.386V10.3217H147.518L147.545 11.3984C147.872 10.7442 148.431 10.2536 149.303 10.2536ZM152.41 8.14108C152.41 8.6181 152.042 8.95883 151.579 8.95883C151.116 8.95883 150.748 8.6181 150.748 8.14108C150.748 7.67769 151.116 7.32334 151.579 7.32334C152.042 7.32334 152.41 7.67769 152.41 8.14108ZM150.938 17V10.3217H152.151V17H150.938ZM157.621 10.2536C159.434 10.2536 160.661 11.5484 160.661 13.5791C160.661 15.6371 159.407 17.0681 157.54 17.0681C156.449 17.0681 155.741 16.6184 155.345 15.9778V19.7258H154.146V10.3217H155.318L155.345 11.4393C155.768 10.7579 156.531 10.2536 157.621 10.2536ZM157.458 16.046C158.657 16.046 159.434 15.051 159.434 13.6064C159.434 12.1889 158.684 11.2758 157.485 11.2758C156.027 11.2758 155.345 12.516 155.345 13.4973V13.7972C155.345 14.8057 156 16.046 157.458 16.046ZM165.921 15.8552L165.975 16.9046C165.757 17 165.348 17.0681 164.926 17.0681C163.372 17.0681 162.445 16.1959 162.445 14.8057V11.3576H161.437V10.3217H162.513V8.33189H163.645V10.3217H165.812V11.3303H163.645V14.8057C163.645 15.4735 163.958 16.0187 165.048 16.0187C165.348 16.0187 165.634 15.9778 165.921 15.8552Z" fill="#f3f4fa"/>+<path d="M93.4642 17.2726L97.4167 7.37785H98.616L94.6908 17.2726H93.4642Z" fill="#b2b8cc"/>+<defs>+<linearGradient id="paint0_linear_1184_806" x1="23.4283" y1="1.77832" x2="3.08491" y2="24.2441" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_806" x1="23.4283" y1="1.77832" x2="3.08491" y2="24.2441" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_806" x1="21.9988" y1="7.74121" x2="12.3565" y2="16.9534" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_806">+<rect width="23.8509" height="23.8509" fill="white" transform="translate(0 0.0742188)"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/crawlee-javascript-light.svg b/website/static/img/crawlee-javascript-light.svgnew file mode 100644index 000000000..2d194f12c--- /dev/null+++ b/website/static/img/crawlee-javascript-light.svg@@ -0,0 +1,67 @@+<svg width="167" height="24" viewBox="0 0 167 24" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_806)">+<path d="M23.0208 22.222H1.25688C0.94552 22.222 0.750917 21.8717 0.906599 21.5895L6.34819 11.7888L11.7898 1.98813C11.9455 1.70838 12.3347 1.70838 12.4903 1.98813L15.9908 8.29326C13.8866 8.81383 12.3274 10.7136 12.3274 12.9783C12.3274 15.6444 14.4875 17.8045 17.1535 17.8045C18.4598 17.8045 19.6469 17.2864 20.5153 16.4398L23.3735 21.5895C23.5292 21.8693 23.3346 22.222 23.0232 22.222H23.0208Z" fill="url(#paint0_linear_1184_806)" stroke="url(#paint1_linear_1184_806)" stroke-width="0.864159" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M21.9988 14.562C21.2398 16.2332 19.5565 17.3935 17.6032 17.3935C14.9371 17.3935 12.777 15.2334 12.777 12.5674C12.777 9.9013 14.9371 7.74121 17.6032 7.74121C18.3524 7.74121 19.0603 7.91149 19.6927 8.21556" stroke="url(#paint2_linear_1184_806)" stroke-width="1.66299" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M33.6469 17.0852C31.1255 17.0852 29.5922 15.3304 29.5922 12.8602C29.5922 10.3388 31.2958 8.56699 33.698 8.56699C36.0831 8.56699 37.3097 9.98101 37.5141 11.855H35.4698C35.3164 10.9862 34.8053 10.2877 33.7491 10.2877C32.5225 10.2877 31.7047 11.2247 31.7047 12.8261C31.7047 14.4275 32.4713 15.3645 33.732 15.3645C34.8394 15.3645 35.3164 14.6831 35.4698 13.8483H37.5141C37.2926 15.6712 36.049 17.0852 33.6469 17.0852ZM43.3607 8.56699C43.5651 8.56699 43.7695 8.60106 43.8717 8.6181L43.8547 10.6625C43.7355 10.6114 43.5992 10.5773 43.2755 10.5773C42.2533 10.5773 41.3844 11.2758 41.2481 12.5365V17H39.1697V8.65217H41.163L41.197 9.99805C41.6059 9.16327 42.2874 8.56699 43.3607 8.56699ZM53.0136 15.2453L53.1499 16.9319C52.8262 17.017 52.4514 17.0852 52.0255 17.0852C51.1226 17.0852 50.4582 16.6252 50.2197 15.7223C49.9471 16.2334 49.2145 17.0852 47.4257 17.0852C45.5687 17.0852 44.5977 16.1652 44.5977 14.8193C44.5977 12.8431 46.5568 12.1957 48.8397 12.1957H50.1345V11.7698C50.1345 10.7988 49.5552 10.1854 48.3968 10.1854C47.4087 10.1854 46.7953 10.6625 46.7102 11.4802H44.8021C44.9384 9.60621 46.3865 8.56699 48.516 8.56699C50.8841 8.56699 52.1618 9.81065 52.1618 11.8891V14.8193C52.1618 15.143 52.3322 15.3304 52.5877 15.3304C52.724 15.3304 52.8262 15.3134 53.0136 15.2453ZM47.9709 15.5349C49.3338 15.5349 50.1345 14.666 50.1345 13.9505V13.4053H48.7375C47.3746 13.4053 46.625 13.8142 46.625 14.5979C46.625 15.1941 47.0509 15.5349 47.9709 15.5349ZM63.9501 8.65217H66.1477L63.7286 17H61.3435L60.0146 12.1446C59.9635 11.9232 59.8784 11.5143 59.8613 11.3439H59.8102C59.7932 11.5143 59.725 11.9232 59.6739 12.1446L58.3451 17H55.96L53.5238 8.65217H55.7215L56.9992 13.9846C57.0844 14.3934 57.1866 14.7342 57.2207 14.9897H57.2718C57.3059 14.7342 57.4081 14.3934 57.5103 13.9846L58.8732 8.65217H60.7983L62.1783 13.9846C62.2805 14.3934 62.3657 14.7342 62.4168 14.9897H62.4679C62.502 14.7342 62.5701 14.4275 62.6894 13.9846L63.9501 8.65217ZM70.3481 15.2453L70.4844 16.9319C70.1607 17.017 69.803 17.0852 69.343 17.0852C68.0993 17.0852 67.4349 16.3185 67.4349 14.9897V4.90417H69.4963V14.8193C69.4963 15.143 69.6837 15.3304 69.9393 15.3304C70.0756 15.3304 70.1607 15.3134 70.3481 15.2453ZM79.5418 12.8772V13.3031H73.562C73.7154 14.666 74.5161 15.4327 75.7086 15.4327C76.9523 15.4327 77.4123 14.7853 77.5485 14.2742H79.5077C79.2181 15.7393 78.0596 17.0852 75.6234 17.0852C73.068 17.0852 71.5177 15.3134 71.5177 12.8431C71.5177 10.3217 73.2554 8.56699 75.6405 8.56699C78.1107 8.56699 79.5418 10.2706 79.5418 12.8772ZM75.6575 10.1684C74.6183 10.1684 73.8176 10.7988 73.5961 12.0254H77.5656C77.3952 10.7476 76.6967 10.1684 75.6575 10.1684ZM88.8419 12.8772V13.3031H82.8622C83.0155 14.666 83.8162 15.4327 85.0087 15.4327C86.2524 15.4327 86.7124 14.7853 86.8487 14.2742H88.8079C88.5182 15.7393 87.3598 17.0852 84.9236 17.0852C82.3681 17.0852 80.8178 15.3134 80.8178 12.8431C80.8178 10.3217 82.5555 8.56699 84.9406 8.56699C87.4109 8.56699 88.8419 10.2706 88.8419 12.8772ZM84.9576 10.1684C83.9184 10.1684 83.1177 10.7988 82.8962 12.0254H86.8657C86.6953 10.7476 85.9969 10.1684 84.9576 10.1684ZM103.155 14.1106C103.168 15.4872 103.918 16.0051 104.749 16.0051C105.69 16.0051 106.344 15.3781 106.344 14.1106V7.65043H107.543V14.0289C107.543 15.8279 106.589 17.0545 104.763 17.0545C102.923 17.0545 101.955 15.9506 101.955 14.1106H103.155ZM115.663 15.9097L115.759 16.9591C115.527 17.0273 115.282 17.0681 114.996 17.0681C114.301 17.0681 113.837 16.632 113.755 15.9778C113.51 16.3594 112.897 17.0681 111.411 17.0681C109.898 17.0681 109.149 16.3185 109.149 15.2555C109.149 13.8108 110.484 13.2111 112.338 13.2111H113.742V12.775C113.742 11.8482 113.169 11.2758 112.093 11.2758C111.139 11.2758 110.553 11.7255 110.471 12.5297H109.299C109.408 11.085 110.539 10.2536 112.174 10.2536C113.933 10.2536 114.941 11.194 114.941 12.8022V15.5281C114.941 15.8143 115.105 15.9778 115.323 15.9778C115.445 15.9778 115.527 15.9642 115.663 15.9097ZM111.657 16.1005C113.006 16.1005 113.742 15.1601 113.742 14.5468V14.0834H112.27C111.016 14.0834 110.375 14.465 110.375 15.201C110.375 15.7598 110.757 16.1005 111.657 16.1005ZM121.256 10.3217H122.482L119.92 17H118.612L116.009 10.3217H117.358L119.075 15.0647C119.171 15.31 119.225 15.5008 119.293 15.7734H119.32C119.375 15.5008 119.429 15.31 119.511 15.0647L121.256 10.3217ZM129.612 15.9097L129.707 16.9591C129.476 17.0273 129.23 17.0681 128.944 17.0681C128.249 17.0681 127.786 16.632 127.704 15.9778C127.459 16.3594 126.845 17.0681 125.36 17.0681C123.847 17.0681 123.097 16.3185 123.097 15.2555C123.097 13.8108 124.433 13.2111 126.287 13.2111H127.69V12.775C127.69 11.8482 127.118 11.2758 126.041 11.2758C125.087 11.2758 124.501 11.7255 124.419 12.5297H123.247C123.356 11.085 124.487 10.2536 126.123 10.2536C127.881 10.2536 128.89 11.194 128.89 12.8022V15.5281C128.89 15.8143 129.053 15.9778 129.271 15.9778C129.394 15.9778 129.476 15.9642 129.612 15.9097ZM125.605 16.1005C126.954 16.1005 127.69 15.1601 127.69 14.5468V14.0834H126.218C124.964 14.0834 124.324 14.465 124.324 15.201C124.324 15.7598 124.706 16.1005 125.605 16.1005ZM132.023 10.0628C132.023 12.5842 137.529 10.7034 137.529 14.5604C137.529 16.0732 136.139 17.0818 134.081 17.0818C131.887 17.0818 130.565 16.0869 130.565 14.3151H131.75C131.75 15.3781 132.595 16.0187 134.176 16.0187C135.512 16.0187 136.33 15.4872 136.316 14.5604C136.316 11.9163 130.81 13.838 130.81 10.09C130.81 8.54996 132.146 7.56866 134.149 7.56866C136.153 7.56866 137.42 8.56359 137.406 10.2672H136.221C136.221 9.13601 135.389 8.63173 134.108 8.63173C132.664 8.63173 132.023 9.14964 132.023 10.0628ZM141.871 17.0681C139.841 17.0681 138.75 15.6507 138.75 13.6881C138.75 11.6846 140.045 10.2536 141.94 10.2536C143.684 10.2536 144.706 11.2349 144.884 12.7613H143.602C143.466 11.93 142.962 11.2758 141.953 11.2758C140.754 11.2758 139.977 12.2162 139.977 13.6609C139.977 15.0919 140.686 16.046 141.926 16.046C142.962 16.046 143.466 15.3918 143.602 14.6013H144.884C144.679 16.0869 143.643 17.0681 141.871 17.0681ZM149.303 10.2536C149.48 10.2536 149.603 10.2809 149.685 10.2945L149.671 11.4802C149.617 11.453 149.494 11.4257 149.221 11.4257C148.376 11.4257 147.586 12.1344 147.586 13.4837V17H146.386V10.3217H147.518L147.545 11.3984C147.872 10.7442 148.431 10.2536 149.303 10.2536ZM152.41 8.14108C152.41 8.6181 152.042 8.95883 151.579 8.95883C151.116 8.95883 150.748 8.6181 150.748 8.14108C150.748 7.67769 151.116 7.32334 151.579 7.32334C152.042 7.32334 152.41 7.67769 152.41 8.14108ZM150.938 17V10.3217H152.151V17H150.938ZM157.621 10.2536C159.434 10.2536 160.661 11.5484 160.661 13.5791C160.661 15.6371 159.407 17.0681 157.54 17.0681C156.449 17.0681 155.741 16.6184 155.345 15.9778V19.7258H154.146V10.3217H155.318L155.345 11.4393C155.768 10.7579 156.531 10.2536 157.621 10.2536ZM157.458 16.046C158.657 16.046 159.434 15.051 159.434 13.6064C159.434 12.1889 158.684 11.2758 157.485 11.2758C156.027 11.2758 155.345 12.516 155.345 13.4973V13.7972C155.345 14.8057 156 16.046 157.458 16.046ZM165.921 15.8552L165.975 16.9046C165.757 17 165.348 17.0681 164.926 17.0681C163.372 17.0681 162.445 16.1959 162.445 14.8057V11.3576H161.437V10.3217H162.513V8.33189H163.645V10.3217H165.812V11.3303H163.645V14.8057C163.645 15.4735 163.958 16.0187 165.048 16.0187C165.348 16.0187 165.634 15.9778 165.921 15.8552Z" fill="#242836"/>+<path d="M93.4642 17.2726L97.4167 7.37785H98.616L94.6908 17.2726H93.4642Z" fill="#555D76"/>+<defs>+<linearGradient id="paint0_linear_1184_806" x1="23.4283" y1="1.77832" x2="3.08491" y2="24.2441" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_806" x1="23.4283" y1="1.77832" x2="3.08491" y2="24.2441" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_806" x1="21.9988" y1="7.74121" x2="12.3565" y2="16.9534" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_806">+<rect width="23.8509" height="23.8509" fill="white" transform="translate(0 0.0742188)"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/crawlee-light.svg b/website/static/img/crawlee-light.svgindex da1498fa9..dc45b948a 100644--- a/website/static/img/crawlee-light.svg+++ b/website/static/img/crawlee-light.svg@@ -1,66 +1,66 @@-<svg width="138" height="42" fill="none" xmlns="http://www.w3.org/2000/svg">- <g clip-path="url(#a)" stroke-linecap="round" stroke-linejoin="round">- <path d="M39.843 34.81h-29.4c-.42 0-.683-.474-.473-.855l7.35-13.24 7.351-13.238a.537.537 0 0 1 .947 0l4.728 8.517a6.521 6.521 0 0 0 1.57 12.848c1.765 0 3.369-.7 4.542-1.843l3.861 6.956c.21.378-.053.854-.473.854h-.003Z" fill="url(#b)" stroke="url(#c)" stroke-width="1.039"/>- <path d="M37.855 25.017a6.519 6.519 0 0 1-5.938 3.825 6.518 6.518 0 0 1-6.52-6.52 6.518 6.518 0 0 1 9.343-5.878" stroke="url(#d)" stroke-width="2"/>- </g>- <path d="M52.868 27.614c3.223 0 4.891-1.897 5.188-4.343h-2.742c-.206 1.12-.846 2.035-2.332 2.035-1.691 0-2.72-1.257-2.72-3.406s1.097-3.406 2.743-3.406c1.417 0 2.103.937 2.308 2.103h2.743c-.274-2.514-1.92-4.411-5.12-4.411-3.222 0-5.508 2.377-5.508 5.76 0 3.314 2.057 5.668 5.44 5.668ZM65.9 16.186c-1.44 0-2.354.8-2.902 1.92l-.046-1.806h-2.674v11.2h2.788v-5.989c.183-1.691 1.349-2.628 2.72-2.628.434 0 .617.046.777.114l.023-2.743a4.258 4.258 0 0 0-.686-.068Zm12.952 8.96a1.483 1.483 0 0 1-.572.114c-.343 0-.571-.251-.571-.686v-3.931c0-2.789-1.715-4.457-4.892-4.457-2.857 0-4.8 1.394-4.983 3.908h2.56c.115-1.097.938-1.737 2.263-1.737 1.555 0 2.332.823 2.332 2.126v.571h-1.737c-3.063 0-5.692.869-5.692 3.52 0 1.806 1.303 3.04 3.794 3.04 2.4 0 3.383-1.143 3.749-1.828.32 1.211 1.211 1.828 2.423 1.828.571 0 1.074-.091 1.508-.205l-.182-2.263Zm-6.766.388c-1.234 0-1.806-.457-1.806-1.257 0-1.051 1.006-1.6 2.834-1.6h1.875v.732c0 .96-1.075 2.125-2.903 2.125ZM93.525 16.3l-1.692 7.154c-.16.595-.251 1.006-.297 1.349h-.069c-.068-.343-.182-.8-.32-1.349L89.296 16.3h-2.583l-1.828 7.154c-.138.549-.275 1.006-.32 1.349h-.069c-.046-.343-.183-.8-.297-1.349L82.485 16.3h-2.949l3.269 11.2h3.2l1.782-6.514c.069-.297.16-.846.183-1.075h.069c.023.229.137.778.206 1.075l1.782 6.514h3.2l3.246-11.2h-2.948Zm8.584 8.846c-.252.091-.366.114-.549.114-.343 0-.594-.251-.594-.686V11.271H98.2v13.532c0 1.783.892 2.811 2.56 2.811.617 0 1.097-.091 1.532-.205l-.183-2.263Zm12.335-3.177c0-3.498-1.92-5.783-5.235-5.783-3.2 0-5.531 2.354-5.531 5.737 0 3.314 2.08 5.691 5.508 5.691 3.269 0 4.823-1.805 5.212-3.771h-2.629c-.183.686-.8 1.554-2.468 1.554-1.6 0-2.675-1.028-2.88-2.857h8.023v-.571Zm-5.212-3.635c1.394 0 2.332.777 2.56 2.492h-5.326c.298-1.646 1.372-2.492 2.766-2.492Zm17.689 3.635c0-3.498-1.92-5.783-5.234-5.783-3.2 0-5.531 2.354-5.531 5.737 0 3.314 2.08 5.691 5.508 5.691 3.269 0 4.823-1.805 5.212-3.771h-2.629c-.183.686-.8 1.554-2.469 1.554-1.6 0-2.674-1.028-2.88-2.857h8.023v-.571Zm-5.211-3.635c1.394 0 2.331.777 2.56 2.492h-5.326c.297-1.646 1.372-2.492 2.766-2.492Z" fill="#242736"/>- <defs>- <linearGradient id="b" x1="40.393" y1="7.193" x2="12.912" y2="37.541" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <linearGradient id="c" x1="40.393" y1="7.193" x2="12.912" y2="37.541" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <linearGradient id="d" x1="37.855" y1="15.803" x2="24.829" y2="28.247" gradientUnits="userSpaceOnUse">- <stop offset=".001" stop-color="#FFB200"/>- <stop offset=".081" stop-color="#FFB100"/>- <stop offset=".156" stop-color="#FFAF02"/>- <stop offset=".225" stop-color="#FEAB04"/>- <stop offset=".291" stop-color="#FDA606"/>- <stop offset=".353" stop-color="#FCA00A"/>- <stop offset=".413" stop-color="#FB980E"/>- <stop offset=".471" stop-color="#FA9013"/>- <stop offset=".529" stop-color="#F98618"/>- <stop offset=".587" stop-color="#F77B1E"/>- <stop offset=".647" stop-color="#F56F24"/>- <stop offset=".709" stop-color="#F3632B"/>- <stop offset=".774" stop-color="#F25532"/>- <stop offset=".844" stop-color="#EF473A"/>- <stop offset=".918" stop-color="#ED3842"/>- <stop offset=".999" stop-color="#EB284B"/>- </linearGradient>- <clipPath id="a">- <path fill="#fff" d="M9.418 6.714h31.45v28.571H9.418z"/>- </clipPath>- </defs>+<svg width="106" height="28" viewBox="0 0 106 28" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_851)">+<path d="M27.0255 26H1.47554C1.11002 26 0.88156 25.5888 1.06432 25.2575L7.45252 13.7519L13.8407 2.2463C14.0235 1.9179 14.4804 1.9179 14.6632 2.2463L18.7725 9.64827C16.3023 10.2594 14.4718 12.4897 14.4718 15.1483C14.4718 18.2782 17.0077 20.8141 20.1375 20.8141C21.671 20.8141 23.0646 20.2058 24.0841 19.212L27.4396 25.2575C27.6223 25.5859 27.3939 26 27.0283 26H27.0255Z" fill="url(#paint0_linear_1184_851)" stroke="url(#paint1_linear_1184_851)" stroke-width="1.01449" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.826 17.0074C24.935 18.9692 22.9588 20.3314 20.6657 20.3314C17.5359 20.3314 15 17.7955 15 14.6657C15 11.5359 17.5359 9 20.6657 9C21.5453 9 22.3763 9.1999 23.1187 9.55686" stroke="url(#paint2_linear_1184_851)" stroke-width="1.95228" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M39.5 20.1C36.54 20.1 34.74 18.04 34.74 15.14C34.74 12.18 36.74 10.1 39.56 10.1C42.36 10.1 43.8 11.76 44.04 13.96H41.64C41.46 12.94 40.86 12.12 39.62 12.12C38.18 12.12 37.22 13.22 37.22 15.1C37.22 16.98 38.12 18.08 39.6 18.08C40.9 18.08 41.46 17.28 41.64 16.3H44.04C43.78 18.44 42.32 20.1 39.5 20.1ZM50.9036 10.1C51.1436 10.1 51.3836 10.14 51.5036 10.16L51.4836 12.56C51.3436 12.5 51.1836 12.46 50.8036 12.46C49.6036 12.46 48.5836 13.28 48.4236 14.76V20H45.9836V10.2H48.3236L48.3636 11.78C48.8436 10.8 49.6436 10.1 50.9036 10.1ZM62.2358 17.94L62.3958 19.92C62.0158 20.02 61.5758 20.1 61.0758 20.1C60.0158 20.1 59.2358 19.56 58.9558 18.5C58.6358 19.1 57.7758 20.1 55.6758 20.1C53.4958 20.1 52.3558 19.02 52.3558 17.44C52.3558 15.12 54.6558 14.36 57.3358 14.36H58.8558V13.86C58.8558 12.72 58.1758 12 56.8158 12C55.6558 12 54.9358 12.56 54.8358 13.52H52.5958C52.7558 11.32 54.4558 10.1 56.9558 10.1C59.7358 10.1 61.2358 11.56 61.2358 14V17.44C61.2358 17.82 61.4358 18.04 61.7358 18.04C61.8958 18.04 62.0158 18.02 62.2358 17.94ZM56.3158 18.28C57.9158 18.28 58.8558 17.26 58.8558 16.42V15.78H57.2158C55.6158 15.78 54.7358 16.26 54.7358 17.18C54.7358 17.88 55.2358 18.28 56.3158 18.28ZM75.0747 10.2H77.6547L74.8147 20H72.0147L70.4547 14.3C70.3947 14.04 70.2947 13.56 70.2747 13.36H70.2147C70.1947 13.56 70.1147 14.04 70.0547 14.3L68.4947 20H65.6947L62.8347 10.2H65.4147L66.9147 16.46C67.0147 16.94 67.1347 17.34 67.1747 17.64H67.2347C67.2747 17.34 67.3947 16.94 67.5147 16.46L69.1147 10.2H71.3747L72.9947 16.46C73.1147 16.94 73.2147 17.34 73.2747 17.64H73.3347C73.3747 17.34 73.4547 16.98 73.5947 16.46L75.0747 10.2ZM82.5858 17.94L82.7458 19.92C82.3658 20.02 81.9458 20.1 81.4058 20.1C79.9458 20.1 79.1658 19.2 79.1658 17.64V5.8H81.5858V17.44C81.5858 17.82 81.8058 18.04 82.1058 18.04C82.2658 18.04 82.3658 18.02 82.5858 17.94ZM93.3788 15.16V15.66H86.3588C86.5388 17.26 87.4788 18.16 88.8788 18.16C90.3388 18.16 90.8788 17.4 91.0388 16.8H93.3388C92.9988 18.52 91.6388 20.1 88.7788 20.1C85.7788 20.1 83.9588 18.02 83.9588 15.12C83.9588 12.16 85.9988 10.1 88.7988 10.1C91.6988 10.1 93.3788 12.1 93.3788 15.16ZM88.8188 11.98C87.5988 11.98 86.6588 12.72 86.3988 14.16H91.0588C90.8588 12.66 90.0388 11.98 88.8188 11.98ZM104.297 15.16V15.66H97.2767C97.4567 17.26 98.3967 18.16 99.7967 18.16C101.257 18.16 101.797 17.4 101.957 16.8H104.257C103.917 18.52 102.557 20.1 99.6967 20.1C96.6967 20.1 94.8767 18.02 94.8767 15.12C94.8767 12.16 96.9167 10.1 99.7167 10.1C102.617 10.1 104.297 12.1 104.297 15.16ZM99.7367 11.98C98.5167 11.98 97.5767 12.72 97.3167 14.16H101.977C101.777 12.66 100.957 11.98 99.7367 11.98Z" fill="#242836"/>+<defs>+<linearGradient id="paint0_linear_1184_851" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_851" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_851" x1="25.826" y1="9" x2="14.5063" y2="19.8147" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_851">+<rect width="28" height="28" fill="white"/>+</clipPath>+</defs></svg>diff --git a/website/static/img/crawlee-logo-monocolor.svg b/website/static/img/crawlee-logo-monocolor.svgnew file mode 100644index 000000000..628e0bae2--- /dev/null+++ b/website/static/img/crawlee-logo-monocolor.svg@@ -0,0 +1,11 @@+<svg xmlns="http://www.w3.org/2000/svg" width="17" height="16" viewBox="0 0 17 16" fill="none">+<g clip-path="url(#clip0_885_92371)">+<path d="M15.9459 14.8569H1.34596C1.13708 14.8569 1.00654 14.6219 1.11098 14.4326L4.76137 7.85796L8.41177 1.28332C8.51621 1.09566 8.7773 1.09566 8.88174 1.28332L11.2299 5.51302C9.81841 5.86223 8.77241 7.13669 8.77241 8.65592C8.77241 10.4444 10.2215 11.8935 12.01 11.8935C12.8862 11.8935 13.6826 11.5459 14.2651 10.978L16.1825 14.4326C16.287 14.6202 16.1564 14.8569 15.9476 14.8569H15.9459Z" fill="var(--color-text)" stroke="var(--color-text)" stroke-width="0.579707" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M15.2605 9.71822C14.7513 10.8393 13.6221 11.6177 12.3118 11.6177C10.5233 11.6177 9.07422 10.1686 9.07422 8.38012C9.07422 6.59164 10.5233 5.14258 12.3118 5.14258C12.8144 5.14258 13.2892 5.25681 13.7135 5.46078" stroke="var(--color-text)" stroke-width="1.11559" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<defs>+<clipPath id="clip0_885_92371">+<rect width="16" height="16" fill="var(--color-text)" transform="translate(0.5)"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/crawlee-python-dark.svg b/website/static/img/crawlee-python-dark.svgnew file mode 100644index 000000000..222d665aa--- /dev/null+++ b/website/static/img/crawlee-python-dark.svg@@ -0,0 +1,67 @@+<svg width="171" height="28" viewBox="0 0 171 28" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_1069)">+<path d="M27.0255 26H1.47554C1.11002 26 0.88156 25.5888 1.06432 25.2575L7.45252 13.7519L13.8407 2.2463C14.0235 1.9179 14.4804 1.9179 14.6632 2.2463L18.7725 9.64827C16.3023 10.2594 14.4718 12.4897 14.4718 15.1483C14.4718 18.2782 17.0077 20.8141 20.1375 20.8141C21.671 20.8141 23.0646 20.2058 24.0841 19.212L27.4396 25.2575C27.6223 25.5859 27.3939 26 27.0283 26H27.0255Z" fill="url(#paint0_linear_1184_1069)" stroke="url(#paint1_linear_1184_1069)" stroke-width="1.01449" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.826 17.0074C24.935 18.9692 22.9588 20.3314 20.6657 20.3314C17.5359 20.3314 15 17.7955 15 14.6657C15 11.5359 17.5359 9 20.6657 9C21.5453 9 22.3763 9.1999 23.1187 9.55686" stroke="url(#paint2_linear_1184_1069)" stroke-width="1.95228" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M39.5 20.1C36.54 20.1 34.74 18.04 34.74 15.14C34.74 12.18 36.74 10.1 39.56 10.1C42.36 10.1 43.8 11.76 44.04 13.96H41.64C41.46 12.94 40.86 12.12 39.62 12.12C38.18 12.12 37.22 13.22 37.22 15.1C37.22 16.98 38.12 18.08 39.6 18.08C40.9 18.08 41.46 17.28 41.64 16.3H44.04C43.78 18.44 42.32 20.1 39.5 20.1ZM50.9036 10.1C51.1436 10.1 51.3836 10.14 51.5036 10.16L51.4836 12.56C51.3436 12.5 51.1836 12.46 50.8036 12.46C49.6036 12.46 48.5836 13.28 48.4236 14.76V20H45.9836V10.2H48.3236L48.3636 11.78C48.8436 10.8 49.6436 10.1 50.9036 10.1ZM62.2358 17.94L62.3958 19.92C62.0158 20.02 61.5758 20.1 61.0758 20.1C60.0158 20.1 59.2358 19.56 58.9558 18.5C58.6358 19.1 57.7758 20.1 55.6758 20.1C53.4958 20.1 52.3558 19.02 52.3558 17.44C52.3558 15.12 54.6558 14.36 57.3358 14.36H58.8558V13.86C58.8558 12.72 58.1758 12 56.8158 12C55.6558 12 54.9358 12.56 54.8358 13.52H52.5958C52.7558 11.32 54.4558 10.1 56.9558 10.1C59.7358 10.1 61.2358 11.56 61.2358 14V17.44C61.2358 17.82 61.4358 18.04 61.7358 18.04C61.8958 18.04 62.0158 18.02 62.2358 17.94ZM56.3158 18.28C57.9158 18.28 58.8558 17.26 58.8558 16.42V15.78H57.2158C55.6158 15.78 54.7358 16.26 54.7358 17.18C54.7358 17.88 55.2358 18.28 56.3158 18.28ZM75.0747 10.2H77.6547L74.8147 20H72.0147L70.4547 14.3C70.3947 14.04 70.2947 13.56 70.2747 13.36H70.2147C70.1947 13.56 70.1147 14.04 70.0547 14.3L68.4947 20H65.6947L62.8347 10.2H65.4147L66.9147 16.46C67.0147 16.94 67.1347 17.34 67.1747 17.64H67.2347C67.2747 17.34 67.3947 16.94 67.5147 16.46L69.1147 10.2H71.3747L72.9947 16.46C73.1147 16.94 73.2147 17.34 73.2747 17.64H73.3347C73.3747 17.34 73.4547 16.98 73.5947 16.46L75.0747 10.2ZM82.5858 17.94L82.7458 19.92C82.3658 20.02 81.9458 20.1 81.4058 20.1C79.9458 20.1 79.1658 19.2 79.1658 17.64V5.8H81.5858V17.44C81.5858 17.82 81.8058 18.04 82.1058 18.04C82.2658 18.04 82.3658 18.02 82.5858 17.94ZM93.3788 15.16V15.66H86.3588C86.5388 17.26 87.4788 18.16 88.8788 18.16C90.3388 18.16 90.8788 17.4 91.0388 16.8H93.3388C92.9988 18.52 91.6388 20.1 88.7788 20.1C85.7788 20.1 83.9588 18.02 83.9588 15.12C83.9588 12.16 85.9988 10.1 88.7988 10.1C91.6988 10.1 93.3788 12.1 93.3788 15.16ZM88.8188 11.98C87.5988 11.98 86.6588 12.72 86.3988 14.16H91.0588C90.8588 12.66 90.0388 11.98 88.8188 11.98ZM104.297 15.16V15.66H97.2767C97.4567 17.26 98.3967 18.16 99.7967 18.16C101.257 18.16 101.797 17.4 101.957 16.8H104.257C103.917 18.52 102.557 20.1 99.6967 20.1C96.6967 20.1 94.8767 18.02 94.8767 15.12C94.8767 12.16 96.9167 10.1 99.7167 10.1C102.617 10.1 104.297 12.1 104.297 15.16ZM99.7367 11.98C98.5167 11.98 97.5767 12.72 97.3167 14.16H101.977C101.777 12.66 100.957 11.98 99.7367 11.98ZM125.409 9.024C127.457 9.024 128.961 10.144 128.961 12.208C128.961 14.272 127.457 15.456 125.409 15.456H122.657V20H121.249V9.024H125.409ZM125.409 14.224C126.817 14.224 127.537 13.344 127.537 12.24C127.537 11.136 126.817 10.256 125.409 10.256H122.657V14.224H125.409ZM135.802 12.16H137.242L134.266 20L133.93 20.88C133.242 22.704 132.49 23.28 131.162 23.28C130.714 23.28 130.25 23.2 129.994 23.088L130.106 21.84C130.378 21.952 130.57 22 130.858 22C131.594 22 132.154 21.84 132.602 20.752L132.794 20.304L129.706 12.16H131.29L133.274 17.728C133.386 18.016 133.45 18.24 133.53 18.56H133.562C133.626 18.24 133.69 18.016 133.786 17.728L135.802 12.16ZM143.096 18.656L143.16 19.888C142.904 20 142.424 20.08 141.928 20.08C140.104 20.08 139.016 19.056 139.016 17.424V13.376H137.832V12.16H139.096V9.824H140.424V12.16H142.968V13.344H140.424V17.424C140.424 18.208 140.792 18.848 142.072 18.848C142.424 18.848 142.76 18.8 143.096 18.656ZM148.49 12.08C150.234 12.08 151.322 13.2 151.322 14.896V20H149.898V15.2C149.898 14 149.258 13.28 148.138 13.28C146.858 13.28 146.01 14.096 146.01 15.44V20H144.586V8.64H145.978V13.312C146.474 12.592 147.29 12.08 148.49 12.08ZM156.962 12.08C159.314 12.08 160.85 13.68 160.85 16.064C160.85 18.432 159.314 20.08 156.962 20.08C154.642 20.08 153.09 18.528 153.09 16.16C153.09 13.776 154.642 12.08 156.962 12.08ZM156.962 13.28C155.506 13.28 154.53 14.448 154.53 16.112C154.53 17.76 155.506 18.88 156.962 18.88C158.45 18.88 159.41 17.76 159.41 16.112C159.41 14.448 158.45 13.28 156.962 13.28ZM166.615 12.08C168.343 12.08 169.447 13.136 169.447 15.056V20H168.023V15.344C168.023 14.016 167.399 13.28 166.231 13.28C164.967 13.28 164.119 14.112 164.119 15.44V20H162.711V12.16H164.039L164.103 13.328C164.599 12.608 165.431 12.08 166.615 12.08Z" fill="#f3f4fa"/>+<path d="M109.723 20.32L114.363 8.704H115.771L111.163 20.32H109.723Z" fill="#b2b8cc"/>+<defs>+<linearGradient id="paint0_linear_1184_1069" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_1069" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_1069" x1="25.826" y1="9" x2="14.5063" y2="19.8147" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_1069">+<rect width="28" height="28" fill="white"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/crawlee-python-light.svg b/website/static/img/crawlee-python-light.svgnew file mode 100644index 000000000..95cdbd04b--- /dev/null+++ b/website/static/img/crawlee-python-light.svg@@ -0,0 +1,67 @@+<svg width="171" height="28" viewBox="0 0 171 28" fill="none" xmlns="http://www.w3.org/2000/svg">+<g clip-path="url(#clip0_1184_1069)">+<path d="M27.0255 26H1.47554C1.11002 26 0.88156 25.5888 1.06432 25.2575L7.45252 13.7519L13.8407 2.2463C14.0235 1.9179 14.4804 1.9179 14.6632 2.2463L18.7725 9.64827C16.3023 10.2594 14.4718 12.4897 14.4718 15.1483C14.4718 18.2782 17.0077 20.8141 20.1375 20.8141C21.671 20.8141 23.0646 20.2058 24.0841 19.212L27.4396 25.2575C27.6223 25.5859 27.3939 26 27.0283 26H27.0255Z" fill="url(#paint0_linear_1184_1069)" stroke="url(#paint1_linear_1184_1069)" stroke-width="1.01449" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.826 17.0074C24.935 18.9692 22.9588 20.3314 20.6657 20.3314C17.5359 20.3314 15 17.7955 15 14.6657C15 11.5359 17.5359 9 20.6657 9C21.5453 9 22.3763 9.1999 23.1187 9.55686" stroke="url(#paint2_linear_1184_1069)" stroke-width="1.95228" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<path d="M39.5 20.1C36.54 20.1 34.74 18.04 34.74 15.14C34.74 12.18 36.74 10.1 39.56 10.1C42.36 10.1 43.8 11.76 44.04 13.96H41.64C41.46 12.94 40.86 12.12 39.62 12.12C38.18 12.12 37.22 13.22 37.22 15.1C37.22 16.98 38.12 18.08 39.6 18.08C40.9 18.08 41.46 17.28 41.64 16.3H44.04C43.78 18.44 42.32 20.1 39.5 20.1ZM50.9036 10.1C51.1436 10.1 51.3836 10.14 51.5036 10.16L51.4836 12.56C51.3436 12.5 51.1836 12.46 50.8036 12.46C49.6036 12.46 48.5836 13.28 48.4236 14.76V20H45.9836V10.2H48.3236L48.3636 11.78C48.8436 10.8 49.6436 10.1 50.9036 10.1ZM62.2358 17.94L62.3958 19.92C62.0158 20.02 61.5758 20.1 61.0758 20.1C60.0158 20.1 59.2358 19.56 58.9558 18.5C58.6358 19.1 57.7758 20.1 55.6758 20.1C53.4958 20.1 52.3558 19.02 52.3558 17.44C52.3558 15.12 54.6558 14.36 57.3358 14.36H58.8558V13.86C58.8558 12.72 58.1758 12 56.8158 12C55.6558 12 54.9358 12.56 54.8358 13.52H52.5958C52.7558 11.32 54.4558 10.1 56.9558 10.1C59.7358 10.1 61.2358 11.56 61.2358 14V17.44C61.2358 17.82 61.4358 18.04 61.7358 18.04C61.8958 18.04 62.0158 18.02 62.2358 17.94ZM56.3158 18.28C57.9158 18.28 58.8558 17.26 58.8558 16.42V15.78H57.2158C55.6158 15.78 54.7358 16.26 54.7358 17.18C54.7358 17.88 55.2358 18.28 56.3158 18.28ZM75.0747 10.2H77.6547L74.8147 20H72.0147L70.4547 14.3C70.3947 14.04 70.2947 13.56 70.2747 13.36H70.2147C70.1947 13.56 70.1147 14.04 70.0547 14.3L68.4947 20H65.6947L62.8347 10.2H65.4147L66.9147 16.46C67.0147 16.94 67.1347 17.34 67.1747 17.64H67.2347C67.2747 17.34 67.3947 16.94 67.5147 16.46L69.1147 10.2H71.3747L72.9947 16.46C73.1147 16.94 73.2147 17.34 73.2747 17.64H73.3347C73.3747 17.34 73.4547 16.98 73.5947 16.46L75.0747 10.2ZM82.5858 17.94L82.7458 19.92C82.3658 20.02 81.9458 20.1 81.4058 20.1C79.9458 20.1 79.1658 19.2 79.1658 17.64V5.8H81.5858V17.44C81.5858 17.82 81.8058 18.04 82.1058 18.04C82.2658 18.04 82.3658 18.02 82.5858 17.94ZM93.3788 15.16V15.66H86.3588C86.5388 17.26 87.4788 18.16 88.8788 18.16C90.3388 18.16 90.8788 17.4 91.0388 16.8H93.3388C92.9988 18.52 91.6388 20.1 88.7788 20.1C85.7788 20.1 83.9588 18.02 83.9588 15.12C83.9588 12.16 85.9988 10.1 88.7988 10.1C91.6988 10.1 93.3788 12.1 93.3788 15.16ZM88.8188 11.98C87.5988 11.98 86.6588 12.72 86.3988 14.16H91.0588C90.8588 12.66 90.0388 11.98 88.8188 11.98ZM104.297 15.16V15.66H97.2767C97.4567 17.26 98.3967 18.16 99.7967 18.16C101.257 18.16 101.797 17.4 101.957 16.8H104.257C103.917 18.52 102.557 20.1 99.6967 20.1C96.6967 20.1 94.8767 18.02 94.8767 15.12C94.8767 12.16 96.9167 10.1 99.7167 10.1C102.617 10.1 104.297 12.1 104.297 15.16ZM99.7367 11.98C98.5167 11.98 97.5767 12.72 97.3167 14.16H101.977C101.777 12.66 100.957 11.98 99.7367 11.98ZM125.409 9.024C127.457 9.024 128.961 10.144 128.961 12.208C128.961 14.272 127.457 15.456 125.409 15.456H122.657V20H121.249V9.024H125.409ZM125.409 14.224C126.817 14.224 127.537 13.344 127.537 12.24C127.537 11.136 126.817 10.256 125.409 10.256H122.657V14.224H125.409ZM135.802 12.16H137.242L134.266 20L133.93 20.88C133.242 22.704 132.49 23.28 131.162 23.28C130.714 23.28 130.25 23.2 129.994 23.088L130.106 21.84C130.378 21.952 130.57 22 130.858 22C131.594 22 132.154 21.84 132.602 20.752L132.794 20.304L129.706 12.16H131.29L133.274 17.728C133.386 18.016 133.45 18.24 133.53 18.56H133.562C133.626 18.24 133.69 18.016 133.786 17.728L135.802 12.16ZM143.096 18.656L143.16 19.888C142.904 20 142.424 20.08 141.928 20.08C140.104 20.08 139.016 19.056 139.016 17.424V13.376H137.832V12.16H139.096V9.824H140.424V12.16H142.968V13.344H140.424V17.424C140.424 18.208 140.792 18.848 142.072 18.848C142.424 18.848 142.76 18.8 143.096 18.656ZM148.49 12.08C150.234 12.08 151.322 13.2 151.322 14.896V20H149.898V15.2C149.898 14 149.258 13.28 148.138 13.28C146.858 13.28 146.01 14.096 146.01 15.44V20H144.586V8.64H145.978V13.312C146.474 12.592 147.29 12.08 148.49 12.08ZM156.962 12.08C159.314 12.08 160.85 13.68 160.85 16.064C160.85 18.432 159.314 20.08 156.962 20.08C154.642 20.08 153.09 18.528 153.09 16.16C153.09 13.776 154.642 12.08 156.962 12.08ZM156.962 13.28C155.506 13.28 154.53 14.448 154.53 16.112C154.53 17.76 155.506 18.88 156.962 18.88C158.45 18.88 159.41 17.76 159.41 16.112C159.41 14.448 158.45 13.28 156.962 13.28ZM166.615 12.08C168.343 12.08 169.447 13.136 169.447 15.056V20H168.023V15.344C168.023 14.016 167.399 13.28 166.231 13.28C164.967 13.28 164.119 14.112 164.119 15.44V20H162.711V12.16H164.039L164.103 13.328C164.599 12.608 165.431 12.08 166.615 12.08Z" fill="#242836"/>+<path d="M109.723 20.32L114.363 8.704H115.771L111.163 20.32H109.723Z" fill="#555D76"/>+<defs>+<linearGradient id="paint0_linear_1184_1069" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint1_linear_1184_1069" x1="27.5039" y1="2" x2="3.62157" y2="28.3739" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<linearGradient id="paint2_linear_1184_1069" x1="25.826" y1="9" x2="14.5063" y2="19.8147" gradientUnits="userSpaceOnUse">+<stop offset="0.000950807" stop-color="#FFB200"/>+<stop offset="0.0814769" stop-color="#FFB100"/>+<stop offset="0.156006" stop-color="#FFAF02"/>+<stop offset="0.225461" stop-color="#FEAB04"/>+<stop offset="0.290764" stop-color="#FDA606"/>+<stop offset="0.352838" stop-color="#FCA00A"/>+<stop offset="0.412606" stop-color="#FB980E"/>+<stop offset="0.47099" stop-color="#FA9013"/>+<stop offset="0.528912" stop-color="#F98618"/>+<stop offset="0.587296" stop-color="#F77B1E"/>+<stop offset="0.647064" stop-color="#F56F24"/>+<stop offset="0.709138" stop-color="#F3632B"/>+<stop offset="0.774441" stop-color="#F25532"/>+<stop offset="0.843896" stop-color="#EF473A"/>+<stop offset="0.918425" stop-color="#ED3842"/>+<stop offset="0.998951" stop-color="#EB284B"/>+</linearGradient>+<clipPath id="clip0_1184_1069">+<rect width="28" height="28" fill="white"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/defaults-dark-icon.svg b/website/static/img/defaults-dark-icon.svgnew file mode 100644index 000000000..8ba564aba--- /dev/null+++ b/website/static/img/defaults-dark-icon.svg@@ -0,0 +1,14 @@+<svg width="41" height="40" viewBox="0 0 41 40" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="-1.56776" y="8.06607" width="33.33" height="33.33" rx="16.665" transform="rotate(-21.3443 -1.56776 8.06607)" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<path d="M4.09246 39.0157L20.2913 6.61803L36.4902 39.0157H4.09246Z" fill="#1A1B21" stroke="#B2B8CC"/>+<g clip-path="url(#clip0_885_94662)">+<path d="M15.9983 33.5C15.9983 33.5 16.1036 33.5 16.2912 33.5C16.4787 33.5 16.7331 33.5 16.9983 33.5H22.9983C23.2635 33.5 23.5179 33.5 23.7054 33.5C23.8929 33.5 23.9983 33.5 23.9983 33.5" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="square" stroke-linejoin="round"/>+<path d="M17.4983 29L19.9983 31.5L22.4983 29" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M19.9983 25.5V31.5" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<defs>+<clipPath id="clip0_885_94662">+<rect width="12" height="12" fill="white" transform="translate(15.9983 22.5)"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/defaults-light-icon.svg b/website/static/img/defaults-light-icon.svgnew file mode 100644index 000000000..f10fcc7c2--- /dev/null+++ b/website/static/img/defaults-light-icon.svg@@ -0,0 +1,14 @@+<svg width="41" height="40" viewBox="0 0 41 40" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="-1.56776" y="8.06607" width="33.33" height="33.33" rx="16.665" transform="rotate(-21.3443 -1.56776 8.06607)" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<path d="M4.09246 39.0157L20.2913 6.61803L36.4902 39.0157H4.09246Z" fill="white" stroke="#555D76"/>+<g clip-path="url(#clip0_885_97537)">+<path d="M15.9983 33.5C15.9983 33.5 16.1036 33.5 16.2912 33.5C16.4787 33.5 16.7331 33.5 16.9983 33.5H22.9983C23.2635 33.5 23.5179 33.5 23.7054 33.5C23.8929 33.5 23.9983 33.5 23.9983 33.5" stroke="#555D76" stroke-width="0.67" stroke-linecap="square" stroke-linejoin="round"/>+<path d="M17.4983 29L19.9983 31.5L22.4983 29" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M19.9983 25.5V31.5" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</g>+<defs>+<clipPath id="clip0_885_97537">+<rect width="12" height="12" fill="white" transform="translate(15.9983 22.5)"/>+</clipPath>+</defs>+</svg>diff --git a/website/static/img/external-link.svg b/website/static/img/external-link.svgnew file mode 100644index 000000000..a3ff6aaa7--- /dev/null+++ b/website/static/img/external-link.svg@@ -0,0 +1,3 @@+<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">+<path fill-rule="evenodd" clip-rule="evenodd" d="M2 1.5C2 1.22386 2.22386 1 2.5 1H6.5C6.77614 1 7 1.22386 7 1.5V5.5C7 5.77614 6.77614 6 6.5 6C6.22386 6 6 5.77614 6 5.5V2.70711L1.85355 6.85355C1.65829 7.04882 1.34171 7.04882 1.14645 6.85355C0.951184 6.65829 0.951184 6.34171 1.14645 6.14645L5.29289 2H2.5C2.22386 2 2 1.77614 2 1.5Z" fill="white" />+</svg>diff --git a/website/static/img/favorite-tools-dark.webp b/website/static/img/favorite-tools-dark.webpnew file mode 100644index 000000000..7aeddcd18Binary files /dev/null and b/website/static/img/favorite-tools-dark.webp differdiff --git a/website/static/img/favorite-tools-light.webp b/website/static/img/favorite-tools-light.webpnew file mode 100644index 000000000..c2596508aBinary files /dev/null and b/website/static/img/favorite-tools-light.webp differdiff --git a/website/static/img/footer-apify-logo.svg b/website/static/img/footer-apify-logo.svgdeleted file mode 100644index 032ec22b3..000000000--- a/website/static/img/footer-apify-logo.svg+++ /dev/null@@ -1,23 +0,0 @@-<svg width="91" height="25" viewBox="0 0 91 25" fill="none" xmlns="http://www.w3.org/2000/svg">- <path- d="M3.13535 2.85082C2.6917 2.91225 2.2645 3.06051 1.87818 3.28712C1.49185 3.51373 1.15397 3.81425 0.883847 4.17151C0.613722 4.52876 0.416649 4.93575 0.30389 5.36921C0.191131 5.80267 0.164898 6.2541 0.226689 6.6977L2.24324 21.0956L10.7262 1.79199L3.13535 2.85082Z"- fill="#97D700"/>- <path- d="M23.6042 14.8474L22.8114 3.77856C22.7446 2.87835 22.3241 2.04122 21.6417 1.45027C20.9593 0.859321 20.0707 0.562647 19.1702 0.625142C19.094 0.625142 19.0177 0.639331 18.9432 0.649972L15.6692 1.10223L22.861 17.2258C23.401 16.5557 23.6665 15.7057 23.6042 14.8474Z"- fill="#71C5E8"/>- <path- d="M5.3363 24.595C5.91693 24.6609 6.50473 24.5754 7.04248 24.3467L19.3919 19.1359L13.5143 5.9707L5.3363 24.595Z"- fill="#FF9013"/>- <path- d="M33.8306 5.30371H37.7325L43.1809 19.9268H39.687L38.6654 16.933H32.7878L31.7626 19.9268H28.3787L33.8306 5.30371ZM33.6533 14.3365H37.7928L35.7922 8.34184H35.7071L33.6533 14.3365Z"- fill="white"/>- <path d="M58.8416 5.30371H62.1439V19.9268H58.8416V5.30371Z" fill="white"/>- <path d="M64.634 5.30371H75.3446V8.00309H67.9435V12.1054H73.9063V14.7373H67.9435V19.9233H64.634V5.30371Z"- fill="white"/>- <path- d="M82.1161 14.3791L76.618 5.30371H80.3656L83.7939 11.3197H83.8791L87.4776 5.30371H91L85.4398 14.3578V19.9268H82.1161V14.3791Z"- fill="white"/>- <path- d="M51.7508 5.30371H44.4579V19.9268H47.7585V15.2925H51.7508C53.0754 15.2925 54.3458 14.7663 55.2824 13.8297C56.219 12.893 56.7452 11.6227 56.7452 10.2981C56.7452 8.9735 56.219 7.70316 55.2824 6.76653C54.3458 5.8299 53.0754 5.30371 51.7508 5.30371ZM51.3855 12.7208H47.7585V7.87539H51.3855C52.028 7.87539 52.6442 8.13064 53.0986 8.58499C53.5529 9.03933 53.8082 9.65555 53.8082 10.2981C53.8082 10.9406 53.5529 11.5569 53.0986 12.0112C52.6442 12.4655 52.028 12.7208 51.3855 12.7208Z"- fill="white"/>-</svg>diff --git a/website/static/img/footer-open-source.svg b/website/static/img/footer-open-source.svgdeleted file mode 100644index e3a59f9ce..000000000--- a/website/static/img/footer-open-source.svg+++ /dev/null@@ -1,8 +0,0 @@-<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">- <path- d="M23.85 21.4261L22.07 18.5161C22.0223 18.4379 21.998 18.3477 22 18.2561V14.1261C22 12.3461 20.29 11.0561 18.48 9.67613C18.4235 9.63333 18.3765 9.5792 18.342 9.51722C18.3076 9.45523 18.2865 9.38675 18.28 9.31613C18.2746 9.24341 18.2842 9.17036 18.3083 9.10153C18.3324 9.0327 18.3704 8.96959 18.42 8.91613C19.2176 8.18843 19.7543 7.21901 19.9476 6.15676C20.1409 5.09451 19.9801 3.99817 19.49 3.03613C19.1249 2.3284 18.5986 1.7164 17.9535 1.24953C17.3084 0.782657 16.5625 0.473992 15.7761 0.348443C14.9897 0.222895 14.1849 0.28398 13.4264 0.526769C12.668 0.769557 11.9773 1.18725 11.41 1.74613L10.35 2.74613C10.2604 2.83348 10.1402 2.88237 10.015 2.88237C9.88984 2.88237 9.76963 2.83348 9.68 2.74613L8.62 1.74613C8.05176 1.18725 7.36022 0.769615 6.60107 0.526864C5.84192 0.284114 5.03637 0.223031 4.24929 0.348535C3.46221 0.474039 2.71561 0.782623 2.0696 1.24944C1.42359 1.71625 0.896234 2.32824 0.529996 3.03613C0.0712553 3.97975 -0.0790193 5.04376 0.100451 6.07751C0.279922 7.11126 0.780043 8.06236 1.53 8.79613L9.29 16.7961C9.39004 16.8988 9.50986 16.98 9.64221 17.035C9.77456 17.09 9.91668 17.1176 10.06 17.1161C10.2048 17.1178 10.3485 17.0904 10.4825 17.0354C10.6165 16.9804 10.7381 16.899 10.84 16.7961C10.9225 16.7101 10.9793 16.6027 11.004 16.4861C11.0288 16.3695 11.0204 16.2483 10.98 16.1361C10.8286 15.6925 10.7607 15.2245 10.78 14.7561C10.8094 14.1798 10.9714 13.6181 11.2535 13.1146C11.5355 12.6112 11.9299 12.1796 12.406 11.8535C12.8821 11.5274 13.427 11.3156 13.9983 11.2345C14.5697 11.1535 15.152 11.2053 15.7 11.3861C15.7886 11.4152 15.8836 11.4184 15.9739 11.3954C16.0643 11.3724 16.1462 11.3241 16.21 11.2561L16.36 11.1061C16.445 11.0188 16.559 10.9655 16.6805 10.9563C16.8021 10.9471 16.9228 10.9826 17.02 11.0561L17.32 11.2861C18.54 12.2061 20.05 13.3561 20.05 14.1361V18.6861C20.0494 18.8734 20.1014 19.057 20.2 19.2161L22.2 22.4761C22.2519 22.6119 22.3328 22.7347 22.437 22.8361C22.5412 22.9374 22.6663 23.0148 22.8034 23.0628C22.9406 23.1109 23.0866 23.1285 23.2313 23.1143C23.376 23.1002 23.5158 23.0547 23.6411 22.981C23.7664 22.9073 23.8741 22.8072 23.9567 22.6876C24.0394 22.568 24.095 22.4319 24.1196 22.2886C24.1442 22.1454 24.1374 21.9985 24.0994 21.8581C24.0615 21.7178 23.9935 21.5875 23.9 21.4761L23.85 21.4261Z"- fill="#B3B8D2"/>- <path- d="M16.09 13.1961C15.6537 12.7991 15.0849 12.5791 14.495 12.5791C13.9051 12.5791 13.3363 12.7991 12.9 13.1961C12.6632 13.4054 12.4704 13.6597 12.333 13.9442C12.1955 14.2288 12.1161 14.5378 12.0993 14.8534C12.0826 15.169 12.1288 15.4847 12.2354 15.7822C12.3419 16.0797 12.5067 16.353 12.72 16.5861L14.35 18.3561C14.4307 18.4514 14.4765 18.5713 14.48 18.6961V19.6961C14.48 20.9961 16 22.2861 16.48 22.6461C16.6349 22.7659 16.8242 22.8325 17.02 22.8361C17.2022 22.8349 17.3795 22.7771 17.5275 22.6707C17.6754 22.5644 17.7867 22.4147 17.846 22.2424C17.9053 22.0701 17.9096 21.8836 17.8584 21.7087C17.8071 21.5339 17.7029 21.3792 17.56 21.2661C16.9911 20.869 16.5388 20.3269 16.25 19.6961V18.1261C16.2512 17.9074 16.1689 17.6964 16.02 17.5361L14.02 15.3561C13.9611 15.2977 13.9155 15.2272 13.8861 15.1496C13.8568 15.072 13.8445 14.9889 13.85 14.9061C13.8613 14.738 13.9364 14.5806 14.06 14.4661C14.1771 14.3621 14.3283 14.3047 14.485 14.3047C14.6416 14.3047 14.7929 14.3621 14.91 14.4661L16.91 16.3261C17.0797 16.4839 17.3052 16.5679 17.5368 16.5594C17.7684 16.551 17.9872 16.4509 18.145 16.2811C18.3028 16.1114 18.3867 15.8859 18.3783 15.6543C18.3698 15.4227 18.2697 15.2039 18.1 15.0461L16.09 13.1961Z"- fill="#B3B8D2"/>-</svg>diff --git a/website/static/img/hearth.svg b/website/static/img/hearth.svgnew file mode 100644index 000000000..007717861--- /dev/null+++ b/website/static/img/hearth.svg@@ -0,0 +1,3 @@+<svg width="14" height="12" viewBox="0 0 14 12" fill="none" xmlns="http://www.w3.org/2000/svg">+<path d="M3.65259 0.0490158C4.24263 -0.0515187 4.84771 -0.0183548 5.42323 0.146063C5.99875 0.310481 6.53004 0.60196 6.97792 0.999016L7.00259 1.02102L7.02526 1.00102C7.45272 0.62588 7.95527 0.346213 8.49937 0.180671C9.04348 0.0151289 9.61663 -0.0324815 10.1806 0.0410159L10.3446 0.0650159C11.0555 0.187755 11.7199 0.500453 12.2676 0.969996C12.8152 1.43954 13.2257 2.04845 13.4556 2.73224C13.6854 3.41603 13.726 4.14925 13.5732 4.85426C13.4203 5.55927 13.0797 6.20982 12.5873 6.73702L12.4673 6.86035L12.4353 6.88768L7.46859 11.807C7.35397 11.9205 7.20212 11.9885 7.04118 11.9986C6.88023 12.0086 6.72109 11.96 6.59326 11.8617L6.53059 11.807L1.53525 6.85902C1.00607 6.34412 0.629722 5.6928 0.447894 4.97719C0.266067 4.26159 0.285856 3.50961 0.505068 2.80456C0.72428 2.0995 1.13436 1.46888 1.6899 0.98253C2.24543 0.496179 2.92474 0.17308 3.65259 0.0490158Z" fill="#B2B8CC"/>+</svg>diff --git a/website/static/img/menu-arrows.svg b/website/static/img/menu-arrows.svgnew file mode 100644index 000000000..0ceeea242--- /dev/null+++ b/website/static/img/menu-arrows.svg@@ -0,0 +1,6 @@+<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">+<g id="Frame">+<path id="Vector" d="M5.3335 5.99967L8.00016 3.33301L10.6668 5.99967" stroke="#B2B8CC" stroke-linecap="round" stroke-linejoin="round"/>+<path id="Vector_2" d="M10.6668 10L8.00016 12.6667L5.3335 10" stroke="#B2B8CC" stroke-linecap="round" stroke-linejoin="round"/>+</g>+</svg>diff --git a/website/static/img/queue-dark-icon.svg b/website/static/img/queue-dark-icon.svgnew file mode 100644index 000000000..4b06fd0c0--- /dev/null+++ b/website/static/img/queue-dark-icon.svg@@ -0,0 +1,8 @@+<svg width="41" height="40" viewBox="0 0 41 40" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="10.1266" cy="29.8737" r="9.79158" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="30.3798" cy="10.1266" r="9.79158" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="20.2531" cy="19.7465" r="14.6899" fill="#1A1B21" stroke="#B2B8CC"/>+<path d="M17.7709 22.9173C17.8676 23.0081 17.9963 23.0567 18.1289 23.0525C18.2614 23.0483 18.3869 22.9916 18.4776 22.8949C18.5684 22.7982 18.617 22.6694 18.6127 22.5368C18.6085 22.4043 18.5518 22.2789 18.4551 22.1881C18.3584 22.0974 18.2297 22.0488 18.0971 22.053C17.9646 22.0572 17.8391 22.1139 17.7484 22.2106C17.6576 22.3073 17.609 22.4361 17.6133 22.5686C17.6175 22.7012 17.6742 22.8266 17.7709 22.9173Z" fill="#B2B8CC" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M20.1673 20.3656C20.264 20.4563 20.3928 20.5049 20.5254 20.5007C20.6579 20.4965 20.7834 20.4398 20.8741 20.3431C20.9648 20.2464 21.0134 20.1176 21.0092 19.9851C21.005 19.8525 20.9483 19.7271 20.8516 19.6364C20.7549 19.5456 20.6261 19.497 20.4936 19.5012C20.3611 19.5054 20.2356 19.5621 20.1449 19.6588C20.0541 19.7555 20.0055 19.8843 20.0097 20.0169C20.014 20.1494 20.0706 20.2748 20.1673 20.3656Z" fill="#B2B8CC" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M22.5609 17.8109C22.6576 17.9016 22.7864 17.9502 22.9189 17.946C23.0515 17.9418 23.1769 17.8851 23.2676 17.7884C23.3584 17.6917 23.407 17.5629 23.4028 17.4304C23.3986 17.2978 23.3419 17.1724 23.2452 17.0817C23.1485 16.9909 23.0197 16.9423 22.8871 16.9465C22.7546 16.9507 22.6292 17.0074 22.5384 17.1041C22.4477 17.2008 22.3991 17.3296 22.4033 17.4622C22.4075 17.5947 22.4642 17.7201 22.5609 17.8109Z" fill="#B2B8CC" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/queue-light-icon.svg b/website/static/img/queue-light-icon.svgnew file mode 100644index 000000000..d31ffda4b--- /dev/null+++ b/website/static/img/queue-light-icon.svg@@ -0,0 +1,8 @@+<svg width="41" height="40" viewBox="0 0 41 40" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="10.1266" cy="29.8737" r="9.79158" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="30.3798" cy="10.1266" r="9.79158" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2 2"/>+<circle cx="20.2531" cy="19.7465" r="14.6899" fill="white" stroke="#555D76"/>+<path d="M17.7709 22.9173C17.8676 23.0081 17.9963 23.0567 18.1289 23.0525C18.2614 23.0483 18.3869 22.9916 18.4776 22.8949C18.5684 22.7982 18.617 22.6694 18.6127 22.5368C18.6085 22.4043 18.5518 22.2789 18.4551 22.1881C18.3584 22.0974 18.2297 22.0488 18.0971 22.053C17.9646 22.0572 17.8391 22.1139 17.7484 22.2106C17.6576 22.3073 17.609 22.4361 17.6133 22.5686C17.6175 22.7012 17.6742 22.8266 17.7709 22.9173Z" fill="#555D76" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M20.1673 20.3656C20.264 20.4563 20.3928 20.5049 20.5254 20.5007C20.6579 20.4965 20.7834 20.4398 20.8741 20.3431C20.9648 20.2464 21.0134 20.1176 21.0092 19.9851C21.005 19.8525 20.9483 19.7271 20.8516 19.6364C20.7549 19.5456 20.6261 19.497 20.4936 19.5012C20.3611 19.5054 20.2356 19.5621 20.1449 19.6588C20.0541 19.7555 20.0055 19.8843 20.0097 20.0169C20.014 20.1494 20.0706 20.2748 20.1673 20.3656Z" fill="#555D76" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M22.5609 17.8109C22.6576 17.9016 22.7864 17.9502 22.9189 17.946C23.0515 17.9418 23.1769 17.8851 23.2676 17.7884C23.3584 17.6917 23.407 17.5629 23.4028 17.4304C23.3986 17.2978 23.3419 17.1724 23.2452 17.0817C23.1485 16.9909 23.0197 16.9423 22.8871 16.9465C22.7546 16.9507 22.6292 17.0074 22.5384 17.1041C22.4477 17.2008 22.3991 17.3296 22.4033 17.4622C22.4075 17.5947 22.4642 17.7201 22.5609 17.8109Z" fill="#555D76" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/routing-dark-icon.svg b/website/static/img/routing-dark-icon.svgnew file mode 100644index 000000000..26585e9b7--- /dev/null+++ b/website/static/img/routing-dark-icon.svg@@ -0,0 +1,8 @@+<svg width="47" height="46" viewBox="0 0 47 46" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="14.525" y="16.4805" width="22.0072" height="22.0072" transform="rotate(-45 14.525 16.4805)" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="7.72205" y="29.5195" width="22.0072" height="22.0072" transform="rotate(-45 7.72205 29.5195)" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="0.91883" y="16.4805" width="22.0072" height="22.0072" transform="rotate(-45 0.91883 16.4805)" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<path d="M7.08514 34.6896L23.284 2.29186L39.4829 34.6896H7.08514Z" fill="#1A1B21" stroke="#B2B8CC"/>+<path d="M20.4504 26.4219L25.8567 21.0156" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.8561 23.2833V21.0156H23.5884" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/routing-light-icon.svg b/website/static/img/routing-light-icon.svgnew file mode 100644index 000000000..0e11097d7--- /dev/null+++ b/website/static/img/routing-light-icon.svg@@ -0,0 +1,8 @@+<svg width="47" height="46" viewBox="0 0 47 46" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="14.5253" y="16.4805" width="22.0072" height="22.0072" transform="rotate(-45 14.5253 16.4805)" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="7.72205" y="29.5195" width="22.0072" height="22.0072" transform="rotate(-45 7.72205 29.5195)" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="0.918952" y="16.4805" width="22.0072" height="22.0072" transform="rotate(-45 0.918952 16.4805)" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<path d="M7.08514 34.6896L23.284 2.29186L39.4829 34.6896H7.08514Z" fill="white" stroke="#555D76"/>+<path d="M20.4507 26.4219L25.8569 21.0156" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+<path d="M25.8565 23.2833V21.0156H23.5887" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/scraping-utils-dark-icon.svg b/website/static/img/scraping-utils-dark-icon.svgnew file mode 100644index 000000000..76db00a84--- /dev/null+++ b/website/static/img/scraping-utils-dark-icon.svg@@ -0,0 +1,5 @@+<svg width="53" height="52" viewBox="0 0 53 52" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="3.24012" y="18.8037" width="22.0072" height="45.0155" transform="rotate(-45 3.24012 18.8037)" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="13.4988" y="13.4971" width="27.3465" height="27.3465" fill="#1A1B21" stroke="#B2B8CC"/>+<path d="M23.0244 26.4987C23.1509 25.5341 23.6245 24.6486 24.3568 24.008C25.089 23.3674 26.0296 23.0157 27.0024 23.0186C27.9753 23.0215 28.9138 23.3789 29.6422 24.0238C30.3706 24.6687 30.8389 25.557 30.9596 26.5224C31.0803 27.4878 30.845 28.464 30.2978 29.2684C29.7505 30.0728 28.9289 30.6502 27.9867 30.8925C27.0445 31.1348 26.0462 31.0254 25.1789 30.5848C24.3115 30.1442 23.6344 29.4025 23.2744 28.4987M23.0244 30.9987V28.4987H25.5244" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/scraping-utils-light-icon.svg b/website/static/img/scraping-utils-light-icon.svgnew file mode 100644index 000000000..10e9c73b0--- /dev/null+++ b/website/static/img/scraping-utils-light-icon.svg@@ -0,0 +1,5 @@+<svg width="53" height="52" viewBox="0 0 53 52" fill="none" xmlns="http://www.w3.org/2000/svg">+<rect x="3.24" y="18.8037" width="22.0072" height="45.0155" transform="rotate(-45 3.24 18.8037)" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2.27 2.27"/>+<rect x="13.4988" y="13.4971" width="27.3465" height="27.3465" fill="white" stroke="#555D76"/>+<path d="M23.0243 26.4987C23.1507 25.5341 23.6244 24.6486 24.3566 24.008C25.0889 23.3674 26.0294 23.0157 27.0023 23.0186C27.9752 23.0215 28.9137 23.3789 29.6421 24.0238C30.3704 24.6687 30.8388 25.557 30.9595 26.5224C31.0801 27.4878 30.8449 28.464 30.2976 29.2684C29.7504 30.0728 28.9288 30.6502 27.9866 30.8925C27.0443 31.1348 26.0461 31.0254 25.1787 30.5848C24.3114 30.1442 23.6343 29.4025 23.2743 28.4987M23.0243 30.9987V28.4987H25.5243" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/smart-proxy-dark.webp b/website/static/img/smart-proxy-dark.webpnew file mode 100644index 000000000..6eec515a0Binary files /dev/null and b/website/static/img/smart-proxy-dark.webp differdiff --git a/website/static/img/smart-proxy-light.webp b/website/static/img/smart-proxy-light.webpnew file mode 100644index 000000000..d38f23f3eBinary files /dev/null and b/website/static/img/smart-proxy-light.webp differdiff --git a/website/static/img/triangles_dark.svg b/website/static/img/triangles_dark.svgnew file mode 100644index 000000000..e1239828e--- /dev/null+++ b/website/static/img/triangles_dark.svg@@ -0,0 +1,38 @@+<svg width="40" height="26" viewBox="0 0 40 26" fill="none" xmlns="http://www.w3.org/2000/svg">+ <path+ d="M14.1096 22.7208C14.1225 22.8058 14.1717 22.8826 14.2464 22.9344C14.3211 22.9862 14.4151 23.0087 14.5079 22.9969L15.9999 22.8055L13.9999 22L14.1096 22.7208Z"+ fill="#474B56" />+ <path+ d="M15.9976 14C16.0047 13.9174 15.9955 13.8337 15.971 13.7572L15.4121 12L13.9999 12.8363L15.9976 14Z"+ fill="#474B56" />+ <path+ d="M14.8567 3.00111L14.1903 3.10091C14.1361 3.10931 14.0857 3.16225 14.0501 3.24815C14.0145 3.33405 13.9967 3.44591 14.0004 3.55927C14.0006 3.56886 14.0011 3.57841 14.0019 3.58785L14.0291 4L14.9999 3.09466C14.9596 3.02675 14.9084 2.99332 14.8567 3.00111Z"+ fill="#474B56" />+ <path+ d="M24.8567 22.0011L24.1903 22.1009C24.1361 22.1093 24.0857 22.1623 24.0501 22.2482C24.0145 22.334 23.9967 22.4459 24.0004 22.5593C24.0006 22.5689 24.0011 22.5784 24.0019 22.5879L24.0291 23L24.9999 22.0947C24.9596 22.0267 24.9084 21.9933 24.8567 22.0011Z"+ fill="#474B56" />+ <path+ d="M24.1096 13.7208C24.1225 13.8058 24.1717 13.8826 24.2464 13.9344C24.3211 13.9862 24.4151 14.0087 24.5079 13.9969L25.9999 13.8055L23.9999 13L24.1096 13.7208Z"+ fill="#474B56" />+ <path+ d="M25.9976 5C26.0047 4.91738 25.9955 4.83375 25.971 4.75722L25.4121 3L23.9999 3.83634L25.9976 5Z"+ fill="#474B56" />+ <path+ d="M35.9976 23C36.0047 22.9174 35.9955 22.8337 35.971 22.7572L35.4121 21L33.9999 21.8363L35.9976 23Z"+ fill="#474B56" />+ <path+ d="M34.8567 12.0011L34.1903 12.1009C34.1361 12.1093 34.0857 12.1623 34.0501 12.2482C34.0145 12.334 33.9967 12.4459 34.0004 12.5593C34.0006 12.5689 34.0011 12.5784 34.0019 12.5879L34.0291 13L34.9999 12.0947C34.9596 12.0267 34.9084 11.9933 34.8567 12.0011Z"+ fill="#474B56" />+ <path+ d="M34.1097 3.72077C34.1225 3.80578 34.1717 3.88263 34.2464 3.93442C34.3211 3.98621 34.4151 4.0087 34.5079 3.99695L35.9999 3.80548L33.9999 3L34.1097 3.72077Z"+ fill="#474B56" />+ <path+ d="M5.99769 23C6.00475 22.9174 5.99557 22.8337 5.97106 22.7572L5.41214 21L4 21.8363L5.99769 23Z"+ fill="#474B56" />+ <path+ d="M4.8568 12.0011L4.19036 12.1009C4.13616 12.1093 4.08575 12.1623 4.05017 12.2482C4.01459 12.334 3.99673 12.4459 4.00049 12.5593C4.00067 12.5689 4.00117 12.5784 4.00199 12.5879L4.02922 13L5 12.0947C4.95964 12.0267 4.90847 11.9933 4.8568 12.0011Z"+ fill="#474B56" />+ <path+ d="M4.10972 3.72077C4.12258 3.80578 4.17176 3.88263 4.24645 3.93442C4.32114 3.98621 4.41521 4.0087 4.508 3.99695L6 3.80548L4 3L4.10972 3.72077Z"+ fill="#474B56" />+</svg>\ No newline at end of filediff --git a/website/static/img/triangles_light.svg b/website/static/img/triangles_light.svgnew file mode 100644index 000000000..bd26bc948--- /dev/null+++ b/website/static/img/triangles_light.svg@@ -0,0 +1,38 @@+<svg width="40" height="26" viewBox="0 0 40 26" fill="none" xmlns="http://www.w3.org/2000/svg">+ <path+ d="M14.1096 22.7208C14.1225 22.8058 14.1717 22.8826 14.2464 22.9344C14.3211 22.9862 14.4151 23.0087 14.5079 22.9969L15.9999 22.8055L13.9999 22L14.1096 22.7208Z"+ fill="#E7EAF1" />+ <path+ d="M15.9976 14C16.0047 13.9174 15.9955 13.8337 15.971 13.7572L15.4121 12L13.9999 12.8363L15.9976 14Z"+ fill="#E7EAF1" />+ <path+ d="M14.8567 3.00111L14.1903 3.10091C14.1361 3.10931 14.0857 3.16225 14.0501 3.24815C14.0145 3.33405 13.9967 3.44591 14.0004 3.55927C14.0006 3.56886 14.0011 3.57841 14.0019 3.58785L14.0291 4L14.9999 3.09466C14.9596 3.02675 14.9084 2.99332 14.8567 3.00111Z"+ fill="#E7EAF1" />+ <path+ d="M24.8567 22.0011L24.1903 22.1009C24.1361 22.1093 24.0857 22.1623 24.0501 22.2482C24.0145 22.334 23.9967 22.4459 24.0004 22.5593C24.0006 22.5689 24.0011 22.5784 24.0019 22.5879L24.0291 23L24.9999 22.0947C24.9596 22.0267 24.9084 21.9933 24.8567 22.0011Z"+ fill="#E7EAF1" />+ <path+ d="M24.1096 13.7208C24.1225 13.8058 24.1717 13.8826 24.2464 13.9344C24.3211 13.9862 24.4151 14.0087 24.5079 13.9969L25.9999 13.8055L23.9999 13L24.1096 13.7208Z"+ fill="#E7EAF1" />+ <path+ d="M25.9976 5C26.0047 4.91738 25.9955 4.83375 25.971 4.75722L25.4121 3L23.9999 3.83634L25.9976 5Z"+ fill="#E7EAF1" />+ <path+ d="M35.9976 23C36.0047 22.9174 35.9955 22.8337 35.971 22.7572L35.4121 21L33.9999 21.8363L35.9976 23Z"+ fill="#E7EAF1" />+ <path+ d="M34.8567 12.0011L34.1903 12.1009C34.1361 12.1093 34.0857 12.1623 34.0501 12.2482C34.0145 12.334 33.9967 12.4459 34.0004 12.5593C34.0006 12.5689 34.0011 12.5784 34.0019 12.5879L34.0291 13L34.9999 12.0947C34.9596 12.0267 34.9084 11.9933 34.8567 12.0011Z"+ fill="#E7EAF1" />+ <path+ d="M34.1097 3.72077C34.1225 3.80578 34.1717 3.88263 34.2464 3.93442C34.3211 3.98621 34.4151 4.0087 34.5079 3.99695L35.9999 3.80548L33.9999 3L34.1097 3.72077Z"+ fill="#E7EAF1" />+ <path+ d="M5.99769 23C6.00475 22.9174 5.99557 22.8337 5.97106 22.7572L5.41214 21L4 21.8363L5.99769 23Z"+ fill="#E7EAF1" />+ <path+ d="M4.8568 12.0011L4.19036 12.1009C4.13616 12.1093 4.08575 12.1623 4.05017 12.2482C4.01459 12.334 3.99673 12.4459 4.00049 12.5593C4.00067 12.5689 4.00117 12.5784 4.00199 12.5879L4.02922 13L5 12.0947C4.95964 12.0267 4.90847 11.9933 4.8568 12.0011Z"+ fill="#E7EAF1" />+ <path+ d="M4.10972 3.72077C4.12258 3.80578 4.17176 3.88263 4.24645 3.93442C4.32114 3.98621 4.41521 4.0087 4.508 3.99695L6 3.80548L4 3L4.10972 3.72077Z"+ fill="#E7EAF1" />+</svg>\ No newline at end of filediff --git a/website/static/img/zero-setup-dark-icon.svg b/website/static/img/zero-setup-dark-icon.svgnew file mode 100644index 000000000..9d70adcd8--- /dev/null+++ b/website/static/img/zero-setup-dark-icon.svg@@ -0,0 +1,6 @@+<svg width="50" height="50" viewBox="0 0 50 50" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="25" cy="25" r="24.665" stroke="#B2B8CC" stroke-width="0.67" stroke-dasharray="2 2"/>+<rect x="12.5" y="12.5" width="25.35" height="25.35" fill="#1A1B21" stroke="#B2B8CC"/>+<path d="M25 20.5C25.96 20.5 26.85 20.8005 27.58 21.313L25 25V20.5Z" fill="#B2B8CC"/>+<path d="M20.5 25C20.5 25.5909 20.6164 26.1761 20.8425 26.7221C21.0687 27.268 21.4002 27.7641 21.818 28.182C22.2359 28.5998 22.732 28.9313 23.2779 29.1575C23.8239 29.3836 24.4091 29.5 25 29.5C25.5909 29.5 26.1761 29.3836 26.7221 29.1575C27.268 28.9313 27.7641 28.5998 28.182 28.182C28.5998 27.7641 28.9313 27.268 29.1575 26.7221C29.3836 26.1761 29.5 25.5909 29.5 25C29.5 23.8065 29.0259 22.6619 28.182 21.818C27.3381 20.9741 26.1935 20.5 25 20.5C23.8065 20.5 22.6619 20.9741 21.818 21.818C20.9741 22.6619 20.5 23.8065 20.5 25Z" stroke="#B2B8CC" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/static/img/zero-setup-light-icon.svg b/website/static/img/zero-setup-light-icon.svgnew file mode 100644index 000000000..97b415804--- /dev/null+++ b/website/static/img/zero-setup-light-icon.svg@@ -0,0 +1,6 @@+<svg width="50" height="50" viewBox="0 0 50 50" fill="none" xmlns="http://www.w3.org/2000/svg">+<circle cx="25" cy="25" r="24.665" stroke="#555D76" stroke-width="0.67" stroke-dasharray="2 2"/>+<rect x="12.5" y="12.5" width="25.35" height="25.35" fill="white" stroke="#555D76"/>+<path d="M25 20.5C25.96 20.5 26.85 20.8005 27.58 21.313L25 25V20.5Z" fill="#555D76"/>+<path d="M20.5 25C20.5 25.5909 20.6164 26.1761 20.8425 26.7221C21.0687 27.268 21.4002 27.7641 21.818 28.182C22.2359 28.5998 22.732 28.9313 23.2779 29.1575C23.8239 29.3836 24.4091 29.5 25 29.5C25.5909 29.5 26.1761 29.3836 26.7221 29.1575C27.268 28.9313 27.7641 28.5998 28.182 28.182C28.5998 27.7641 28.9313 27.268 29.1575 26.7221C29.3836 26.1761 29.5 25.5909 29.5 25C29.5 23.8065 29.0259 22.6619 28.182 21.818C27.3381 20.9741 26.1935 20.5 25 20.5C23.8065 20.5 22.6619 20.9741 21.818 21.818C20.9741 22.6619 20.5 23.8065 20.5 25Z" stroke="#555D76" stroke-width="0.67" stroke-linecap="round" stroke-linejoin="round"/>+</svg>diff --git a/website/tools/website_gif/website_gif.mjs b/website/tools/website_gif/website_gif.mjsindex 4c4f48313..5c5a0fc54 100644--- a/website/tools/website_gif/website_gif.mjs+++ b/website/tools/website_gif/website_gif.mjs@@ -35,23 +35,23 @@ await crawler.run([uniqueKey: 'dark-start'},{- url: 'https://crawlee.dev/docs/quick-start',+ url: 'https://crawlee.dev/js/docs/quick-start',uniqueKey: 'dark-1'},{- url: 'https://crawlee.dev/docs/introduction/setting-up',+ url: 'https://crawlee.dev/js/docs/introduction/setting-up',uniqueKey: 'dark-2'},{- url: 'https://crawlee.dev/docs/introduction/first-crawler',+ url: 'https://crawlee.dev/js/docs/introduction/first-crawler',uniqueKey: 'dark-3'},{- url: 'https://crawlee.dev/docs/introduction/adding-urls',+ url: 'https://crawlee.dev/js/docs/introduction/adding-urls',uniqueKey: 'dark-4'},{- url: 'https://crawlee.dev/docs/introduction/real-world-project',+ url: 'https://crawlee.dev/js/docs/introduction/real-world-project',uniqueKey: 'dark-5'},@@ -62,23 +62,23 @@ await crawler.run([uniqueKey: 'light th-start'},{- url: 'https://crawlee.dev/docs/quick-start',+ url: 'https://crawlee.dev/js/docs/quick-start',uniqueKey: 'light th-1'},{- url: 'https://crawlee.dev/docs/introduction/setting-up',+ url: 'https://crawlee.dev/js/docs/introduction/setting-up',uniqueKey: 'light th-2'},{- url: 'https://crawlee.dev/docs/introduction/first-crawler',+ url: 'https://crawlee.dev/js/docs/introduction/first-crawler',uniqueKey: 'light th-3'},{- url: 'https://crawlee.dev/docs/introduction/adding-urls',+ url: 'https://crawlee.dev/js/docs/introduction/adding-urls',uniqueKey: 'light th-4'},{- url: 'https://crawlee.dev/docs/introduction/real-world-project',+ url: 'https://crawlee.dev/js/docs/introduction/real-world-project',uniqueKey: 'light th-5'}]);diff --git a/website/versioned_docs/version-3.0/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.0/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.0/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.0/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.0/examples/http_crawler.ts b/website/versioned_docs/version-3.0/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.0/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.0/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.0/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.0/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.0/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.0/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.0/guides/docker_browser_js.txt b/website/versioned_docs/version-3.0/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.0/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.0/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.0/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.0/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.0/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.0/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.0/guides/docker_node_js.txt b/website/versioned_docs/version-3.0/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.0/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.0/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.0/guides/docker_node_ts.txt b/website/versioned_docs/version-3.0/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.0/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.0/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.0/guides/request_storage.mdx b/website/versioned_docs/version-3.0/guides/request_storage.mdxindex a655322ea..829177e5b 100644--- a/website/versioned_docs/version-3.0/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.0/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.0/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.0/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.0/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.0/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.0/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.1/api-typedoc.json b/website/versioned_docs/version-3.1/api-typedoc.jsonindex 0fcecdb82..f2639a537 100644--- a/website/versioned_docs/version-3.1/api-typedoc.json+++ b/website/versioned_docs/version-3.1/api-typedoc.json@@ -3834,7 +3834,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -4003,7 +4003,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -10166,7 +10166,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -10285,7 +10285,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -94354,7 +94354,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -94471,7 +94471,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -133419,7 +133419,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -133528,7 +133528,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -150109,7 +150109,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -150226,7 +150226,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -156157,7 +156157,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -181606,7 +181606,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -182295,7 +182295,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -194121,7 +194121,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -194884,7 +194884,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.1/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.1/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.1/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.1/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.1/examples/http_crawler.ts b/website/versioned_docs/version-3.1/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.1/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.1/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.1/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.1/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.1/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.1/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.1/guides/docker_browser_js.txt b/website/versioned_docs/version-3.1/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.1/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.1/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.1/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.1/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.1/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.1/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.1/guides/docker_node_js.txt b/website/versioned_docs/version-3.1/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.1/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.1/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.1/guides/docker_node_ts.txt b/website/versioned_docs/version-3.1/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.1/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.1/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.1/guides/request_storage.mdx b/website/versioned_docs/version-3.1/guides/request_storage.mdxindex a655322ea..829177e5b 100644--- a/website/versioned_docs/version-3.1/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.1/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.1/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.1/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.1/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.1/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.1/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.1/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.1/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.1/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.1/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.1/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.1/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.1/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.1/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.1/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.1/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.1/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.10/api-typedoc.json b/website/versioned_docs/version-3.10/api-typedoc.jsonindex d08ed4bbf..7ebff94f7 100644--- a/website/versioned_docs/version-3.10/api-typedoc.json+++ b/website/versioned_docs/version-3.10/api-typedoc.json@@ -5793,7 +5793,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -5838,7 +5838,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -7102,7 +7102,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -16714,7 +16714,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -16759,7 +16759,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -17094,7 +17094,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17577,7 +17577,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -109969,7 +109969,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -110386,7 +110386,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -167212,7 +167212,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -167257,7 +167257,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -167592,7 +167592,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -175847,7 +175847,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -192563,7 +192563,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -192960,7 +192960,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -196168,7 +196168,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -196565,7 +196565,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -206323,7 +206323,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -206741,7 +206741,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -216464,7 +216464,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -216882,7 +216882,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -248542,7 +248542,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -267052,7 +267052,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -267097,7 +267097,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -268091,7 +268091,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -283008,7 +283008,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -283053,7 +283053,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -284135,7 +284135,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -295013,7 +295013,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -299763,7 +299763,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -299811,7 +299811,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.10/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.10/examples/cheerio_crawler.tsindex 7a34547d1..04f41981b 100644--- a/website/versioned_docs/version-3.10/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.10/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.10/examples/http_crawler.ts b/website/versioned_docs/version-3.10/examples/http_crawler.tsindex ef5492340..add990c59 100644--- a/website/versioned_docs/version-3.10/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.10/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.10/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.10/examples/jsdom_crawler.tsindex be588156c..8d1ab514c 100644--- a/website/versioned_docs/version-3.10/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.10/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.10/experiments/request_locking.mdx b/website/versioned_docs/version-3.10/experiments/request_locking.mdxindex 224b59ec7..46ad112f4 100644--- a/website/versioned_docs/version-3.10/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.10/experiments/request_locking.mdx@@ -82,8 +82,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.10/guides/docker_browser_js.txt b/website/versioned_docs/version-3.10/guides/docker_browser_js.txtindex 6e804b93a..76848bac3 100644--- a/website/versioned_docs/version-3.10/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.10/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20diff --git a/website/versioned_docs/version-3.10/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.10/guides/docker_browser_ts.txtindex 1fe6784a4..513bccfc1 100644--- a/website/versioned_docs/version-3.10/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.10/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20 AS builderdiff --git a/website/versioned_docs/version-3.10/guides/docker_node_js.txt b/website/versioned_docs/version-3.10/guides/docker_node_js.txtindex 4c8d11fc3..50cecc043 100644--- a/website/versioned_docs/version-3.10/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.10/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20diff --git a/website/versioned_docs/version-3.10/guides/docker_node_ts.txt b/website/versioned_docs/version-3.10/guides/docker_node_ts.txtindex 995a3d815..30d85ce51 100644--- a/website/versioned_docs/version-3.10/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.10/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20 AS builderdiff --git a/website/versioned_docs/version-3.10/guides/request_storage.mdx b/website/versioned_docs/version-3.10/guides/request_storage.mdxindex 279a504c9..8da5489b5 100644--- a/website/versioned_docs/version-3.10/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.10/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.10/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.10/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.10/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.10/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.10/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.10/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.10/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.10/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.10/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.10/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.10/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.10/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.10/upgrading/upgrading_v3.mdindex aa62f6c08..39b6091c9 100644--- a/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.10/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.11/api-typedoc.json b/website/versioned_docs/version-3.11/api-typedoc.jsonindex 5f377dfbe..109930247 100644--- a/website/versioned_docs/version-3.11/api-typedoc.json+++ b/website/versioned_docs/version-3.11/api-typedoc.json@@ -5919,7 +5919,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -5964,7 +5964,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -7258,7 +7258,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17085,7 +17085,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -17130,7 +17130,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -17465,7 +17465,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17948,7 +17948,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -112411,7 +112411,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -112864,7 +112864,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -171548,7 +171548,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -171593,7 +171593,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -171928,7 +171928,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -180984,7 +180984,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -198436,7 +198436,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -198898,7 +198898,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -202224,7 +202224,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -202686,7 +202686,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -212915,7 +212915,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -213369,7 +213369,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -223680,7 +223680,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -224134,7 +224134,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -256700,7 +256700,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -276078,7 +276078,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -276123,7 +276123,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -277153,7 +277153,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -296175,7 +296175,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -296220,7 +296220,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -297338,7 +297338,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -308408,7 +308408,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -313496,7 +313496,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -313544,7 +313544,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.11/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.11/examples/cheerio_crawler.tsindex 7a34547d1..04f41981b 100644--- a/website/versioned_docs/version-3.11/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.11/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.11/examples/http_crawler.ts b/website/versioned_docs/version-3.11/examples/http_crawler.tsindex ef5492340..add990c59 100644--- a/website/versioned_docs/version-3.11/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.11/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.11/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.11/examples/jsdom_crawler.tsindex f1760044f..6db02c4ef 100644--- a/website/versioned_docs/version-3.11/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.11/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.11/experiments/request_locking.mdx b/website/versioned_docs/version-3.11/experiments/request_locking.mdxindex 224b59ec7..46ad112f4 100644--- a/website/versioned_docs/version-3.11/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.11/experiments/request_locking.mdx@@ -82,8 +82,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.11/guides/docker_browser_js.txt b/website/versioned_docs/version-3.11/guides/docker_browser_js.txtindex 6e804b93a..76848bac3 100644--- a/website/versioned_docs/version-3.11/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.11/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20diff --git a/website/versioned_docs/version-3.11/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.11/guides/docker_browser_ts.txtindex 1fe6784a4..513bccfc1 100644--- a/website/versioned_docs/version-3.11/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.11/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20 AS builderdiff --git a/website/versioned_docs/version-3.11/guides/docker_node_js.txt b/website/versioned_docs/version-3.11/guides/docker_node_js.txtindex 4c8d11fc3..50cecc043 100644--- a/website/versioned_docs/version-3.11/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.11/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20diff --git a/website/versioned_docs/version-3.11/guides/docker_node_ts.txt b/website/versioned_docs/version-3.11/guides/docker_node_ts.txtindex 995a3d815..30d85ce51 100644--- a/website/versioned_docs/version-3.11/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.11/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20 AS builderdiff --git a/website/versioned_docs/version-3.11/guides/request_storage.mdx b/website/versioned_docs/version-3.11/guides/request_storage.mdxindex 279a504c9..8da5489b5 100644--- a/website/versioned_docs/version-3.11/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.11/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdxindex 39c347510..cee0f83a9 100644--- a/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx@@ -10,7 +10,7 @@ import ApiLink from '@site/src/components/ApiLink';import WebServerSource from '!!raw-loader!./web-server.mjs';-Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.+Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.We will build a simple HTTP server that receives a page URL and returns the page title in the response. We will base this guide on the approach used in [Apify's Super Scraper API repository](https://github.com/apify/super-scraper) which maps incoming HTTP requests to Crawlee <ApiLink to="core/class/Request">Request</ApiLink>.diff --git a/website/versioned_docs/version-3.11/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.11/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.11/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.11/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.11/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.11/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.11/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.11/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.11/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.11/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.11/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.11/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.11/upgrading/upgrading_v3.mdindex aa62f6c08..39b6091c9 100644--- a/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.11/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.12/api-typedoc.json b/website/versioned_docs/version-3.12/api-typedoc.jsonindex d9ece17ec..dcfd095bc 100644--- a/website/versioned_docs/version-3.12/api-typedoc.json+++ b/website/versioned_docs/version-3.12/api-typedoc.json@@ -6140,7 +6140,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6185,7 +6185,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -7487,7 +7487,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17622,7 +17622,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -17667,7 +17667,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -18002,7 +18002,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -18485,7 +18485,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -119449,7 +119449,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -119902,7 +119902,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -179417,7 +179417,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -179462,7 +179462,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -179797,7 +179797,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -191541,7 +191541,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -210208,7 +210208,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -210670,7 +210670,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -214046,7 +214046,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -214508,7 +214508,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -224985,7 +224985,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -225439,7 +225439,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -236006,7 +236006,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -236460,7 +236460,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -271503,7 +271503,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -292657,7 +292657,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -292702,7 +292702,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -293732,7 +293732,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -313384,7 +313384,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -313429,7 +313429,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -314547,7 +314547,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -325746,7 +325746,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -331094,7 +331094,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n> Crawlee for Python is open for early adopters. 🐍 [👉 Checkout the source code 👈](https://github.com/apify/crawlee-python).\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n> Crawlee for Python is open for early adopters. 🐍 [👉 Checkout the source code 👈](https://github.com/apify/crawlee-python).\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -331142,7 +331142,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.12/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.12/examples/cheerio_crawler.tsindex 7a34547d1..04f41981b 100644--- a/website/versioned_docs/version-3.12/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.12/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.12/examples/http_crawler.ts b/website/versioned_docs/version-3.12/examples/http_crawler.tsindex ef5492340..add990c59 100644--- a/website/versioned_docs/version-3.12/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.12/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.12/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.12/examples/jsdom_crawler.tsindex f1760044f..6db02c4ef 100644--- a/website/versioned_docs/version-3.12/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.12/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.12/experiments/request_locking.mdx b/website/versioned_docs/version-3.12/experiments/request_locking.mdxindex 224b59ec7..46ad112f4 100644--- a/website/versioned_docs/version-3.12/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.12/experiments/request_locking.mdx@@ -82,8 +82,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.12/guides/docker_browser_js.txt b/website/versioned_docs/version-3.12/guides/docker_browser_js.txtindex 6e804b93a..76848bac3 100644--- a/website/versioned_docs/version-3.12/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.12/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20diff --git a/website/versioned_docs/version-3.12/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.12/guides/docker_browser_ts.txtindex 1fe6784a4..513bccfc1 100644--- a/website/versioned_docs/version-3.12/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.12/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20 AS builderdiff --git a/website/versioned_docs/version-3.12/guides/docker_node_js.txt b/website/versioned_docs/version-3.12/guides/docker_node_js.txtindex 4c8d11fc3..50cecc043 100644--- a/website/versioned_docs/version-3.12/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.12/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20diff --git a/website/versioned_docs/version-3.12/guides/docker_node_ts.txt b/website/versioned_docs/version-3.12/guides/docker_node_ts.txtindex 995a3d815..30d85ce51 100644--- a/website/versioned_docs/version-3.12/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.12/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20 AS builderdiff --git a/website/versioned_docs/version-3.12/guides/request_storage.mdx b/website/versioned_docs/version-3.12/guides/request_storage.mdxindex 279a504c9..8da5489b5 100644--- a/website/versioned_docs/version-3.12/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.12/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdxindex 39c347510..cee0f83a9 100644--- a/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx@@ -10,7 +10,7 @@ import ApiLink from '@site/src/components/ApiLink';import WebServerSource from '!!raw-loader!./web-server.mjs';-Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.+Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.We will build a simple HTTP server that receives a page URL and returns the page title in the response. We will base this guide on the approach used in [Apify's Super Scraper API repository](https://github.com/apify/super-scraper) which maps incoming HTTP requests to Crawlee <ApiLink to="core/class/Request">Request</ApiLink>.diff --git a/website/versioned_docs/version-3.12/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.12/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.12/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.12/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.12/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.12/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.12/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.12/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.12/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.12/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.12/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.12/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.12/upgrading/upgrading_v3.mdindex aa62f6c08..39b6091c9 100644--- a/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.12/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.13/api-typedoc.json b/website/versioned_docs/version-3.13/api-typedoc.jsonindex 48b7c99c6..ce0d512c1 100644--- a/website/versioned_docs/version-3.13/api-typedoc.json+++ b/website/versioned_docs/version-3.13/api-typedoc.json@@ -6239,7 +6239,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6284,7 +6284,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -7594,7 +7594,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17924,7 +17924,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -17971,7 +17971,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -18314,7 +18314,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -18734,7 +18734,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -120078,7 +120078,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -120539,7 +120539,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -181358,7 +181358,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -181403,7 +181403,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -181742,7 +181742,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -193653,7 +193653,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -212709,7 +212709,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -213183,7 +213183,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -216633,7 +216633,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -217107,7 +217107,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -227820,7 +227820,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -228282,7 +228282,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -239095,7 +239095,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -239557,7 +239557,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -275096,7 +275096,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -296322,7 +296322,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -296369,7 +296369,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -297548,7 +297548,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -317457,7 +317457,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -317504,7 +317504,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -318658,7 +318658,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -329930,7 +329930,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -335516,7 +335516,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n> Crawlee for Python is open for early adopters. 🐍 [👉 Checkout the source code 👈](https://github.com/apify/crawlee-python).\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n> Crawlee for Python is open for early adopters. 🐍 [👉 Checkout the source code 👈](https://github.com/apify/crawlee-python).\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -335564,7 +335564,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n### Installing pre-release versions\n\nWe provide automated beta builds for every merged code change in Crawlee. You can find them in the npm [list of releases](https://www.npmjs.com/package/crawlee?activeTab=versions). If you want to test new features or bug fixes before we release them, feel free to install a beta build like this:\n\n"+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n### Installing pre-release versions\n\nWe provide automated beta builds for every merged code change in Crawlee. You can find them in the npm [list of releases](https://www.npmjs.com/package/crawlee?activeTab=versions). If you want to test new features or bug fixes before we release them, feel free to install a beta build like this:\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.13/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.13/examples/cheerio_crawler.tsindex 7a34547d1..04f41981b 100644--- a/website/versioned_docs/version-3.13/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.13/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.13/examples/http_crawler.ts b/website/versioned_docs/version-3.13/examples/http_crawler.tsindex ef5492340..add990c59 100644--- a/website/versioned_docs/version-3.13/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.13/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.13/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.13/examples/jsdom_crawler.tsindex f1760044f..6db02c4ef 100644--- a/website/versioned_docs/version-3.13/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.13/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.13/experiments/request_locking.mdx b/website/versioned_docs/version-3.13/experiments/request_locking.mdxindex 224b59ec7..46ad112f4 100644--- a/website/versioned_docs/version-3.13/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.13/experiments/request_locking.mdx@@ -82,8 +82,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.13/guides/docker_browser_js.txt b/website/versioned_docs/version-3.13/guides/docker_browser_js.txtindex 6e804b93a..76848bac3 100644--- a/website/versioned_docs/version-3.13/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.13/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20diff --git a/website/versioned_docs/version-3.13/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.13/guides/docker_browser_ts.txtindex 1fe6784a4..513bccfc1 100644--- a/website/versioned_docs/version-3.13/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.13/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:20 AS builderdiff --git a/website/versioned_docs/version-3.13/guides/docker_node_js.txt b/website/versioned_docs/version-3.13/guides/docker_node_js.txtindex 4c8d11fc3..50cecc043 100644--- a/website/versioned_docs/version-3.13/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.13/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20diff --git a/website/versioned_docs/version-3.13/guides/docker_node_ts.txt b/website/versioned_docs/version-3.13/guides/docker_node_ts.txtindex 995a3d815..30d85ce51 100644--- a/website/versioned_docs/version-3.13/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.13/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:20 AS builderdiff --git a/website/versioned_docs/version-3.13/guides/request_storage.mdx b/website/versioned_docs/version-3.13/guides/request_storage.mdxindex 279a504c9..8da5489b5 100644--- a/website/versioned_docs/version-3.13/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.13/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdxindex 39c347510..cee0f83a9 100644--- a/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx@@ -10,7 +10,7 @@ import ApiLink from '@site/src/components/ApiLink';import WebServerSource from '!!raw-loader!./web-server.mjs';-Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.+Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.We will build a simple HTTP server that receives a page URL and returns the page title in the response. We will base this guide on the approach used in [Apify's Super Scraper API repository](https://github.com/apify/super-scraper) which maps incoming HTTP requests to Crawlee <ApiLink to="core/class/Request">Request</ApiLink>.diff --git a/website/versioned_docs/version-3.13/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.13/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.13/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.13/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.13/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.13/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.13/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.13/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.13/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.13/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.13/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.13/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.13/upgrading/upgrading_v3.mdindex aa62f6c08..39b6091c9 100644--- a/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.13/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.2/api-typedoc.json b/website/versioned_docs/version-3.2/api-typedoc.jsonindex 62298be24..f61c699f0 100644--- a/website/versioned_docs/version-3.2/api-typedoc.json+++ b/website/versioned_docs/version-3.2/api-typedoc.json@@ -3818,7 +3818,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -3987,7 +3987,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -10150,7 +10150,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -10269,7 +10269,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -10640,7 +10640,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -95217,7 +95217,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -95334,7 +95334,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -134805,7 +134805,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -134914,7 +134914,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -151310,7 +151310,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -151427,7 +151427,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -157406,7 +157406,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -183321,7 +183321,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -184010,7 +184010,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -195960,7 +195960,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -196723,7 +196723,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.2/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.2/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.2/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.2/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.2/examples/http_crawler.ts b/website/versioned_docs/version-3.2/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.2/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.2/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.2/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.2/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.2/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.2/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.2/guides/docker_browser_js.txt b/website/versioned_docs/version-3.2/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.2/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.2/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.2/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.2/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.2/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.2/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.2/guides/docker_node_js.txt b/website/versioned_docs/version-3.2/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.2/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.2/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.2/guides/docker_node_ts.txt b/website/versioned_docs/version-3.2/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.2/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.2/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.2/guides/request_storage.mdx b/website/versioned_docs/version-3.2/guides/request_storage.mdxindex a655322ea..829177e5b 100644--- a/website/versioned_docs/version-3.2/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.2/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.2/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.2/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.2/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.2/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.2/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.2/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.2/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.2/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.2/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.2/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.2/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.2/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.2/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.2/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.2/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.2/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.3/api-typedoc.json b/website/versioned_docs/version-3.3/api-typedoc.jsonindex b927f5660..3f1eec5a4 100644--- a/website/versioned_docs/version-3.3/api-typedoc.json+++ b/website/versioned_docs/version-3.3/api-typedoc.json@@ -3888,7 +3888,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -4383,7 +4383,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -10622,7 +10622,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -10741,7 +10741,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -11112,7 +11112,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -94278,7 +94278,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -94395,7 +94395,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -133868,7 +133868,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -133977,7 +133977,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -151103,7 +151103,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -151220,7 +151220,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -157276,7 +157276,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -183381,7 +183381,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -184070,7 +184070,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -195633,7 +195633,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -196396,7 +196396,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.3/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.3/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.3/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.3/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.3/examples/http_crawler.ts b/website/versioned_docs/version-3.3/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.3/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.3/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.3/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.3/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.3/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.3/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.3/guides/docker_browser_js.txt b/website/versioned_docs/version-3.3/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.3/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.3/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.3/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.3/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.3/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.3/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.3/guides/docker_node_js.txt b/website/versioned_docs/version-3.3/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.3/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.3/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.3/guides/docker_node_ts.txt b/website/versioned_docs/version-3.3/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.3/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.3/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.3/guides/request_storage.mdx b/website/versioned_docs/version-3.3/guides/request_storage.mdxindex a655322ea..829177e5b 100644--- a/website/versioned_docs/version-3.3/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.3/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.3/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.3/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.3/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.3/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.3/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.3/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.3/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.3/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.3/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.3/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.3/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.3/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.3/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.3/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.3/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.3/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.4/api-typedoc.json b/website/versioned_docs/version-3.4/api-typedoc.jsonindex 6ca1d1fa2..6af245123 100644--- a/website/versioned_docs/version-3.4/api-typedoc.json+++ b/website/versioned_docs/version-3.4/api-typedoc.json@@ -3888,7 +3888,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -4907,7 +4907,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -11398,7 +11398,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -11517,7 +11517,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -11888,7 +11888,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -96006,7 +96006,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -96189,7 +96189,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -135863,7 +135863,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -135972,7 +135972,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -153962,7 +153962,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -154125,7 +154125,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -160286,7 +160286,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -160470,7 +160470,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -186922,7 +186922,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -187611,7 +187611,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -199239,7 +199239,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -200002,7 +200002,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.4/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.4/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.4/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.4/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.4/examples/http_crawler.ts b/website/versioned_docs/version-3.4/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.4/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.4/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.4/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.4/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.4/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.4/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.4/guides/docker_browser_js.txt b/website/versioned_docs/version-3.4/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.4/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.4/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.4/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.4/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.4/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.4/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.4/guides/docker_node_js.txt b/website/versioned_docs/version-3.4/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.4/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.4/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.4/guides/docker_node_ts.txt b/website/versioned_docs/version-3.4/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.4/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.4/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.4/guides/request_storage.mdx b/website/versioned_docs/version-3.4/guides/request_storage.mdxindex a655322ea..829177e5b 100644--- a/website/versioned_docs/version-3.4/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.4/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.4/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.4/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.4/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.4/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.4/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.4/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.4/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.4/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.4/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.4/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.4/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.4/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.4/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.4/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.4/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.4/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.5/api-typedoc.json b/website/versioned_docs/version-3.5/api-typedoc.jsonindex dcea7946e..7b9fa9319 100644--- a/website/versioned_docs/version-3.5/api-typedoc.json+++ b/website/versioned_docs/version-3.5/api-typedoc.json@@ -4229,7 +4229,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -5248,7 +5248,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -12628,7 +12628,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -12747,7 +12747,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -13118,7 +13118,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -97750,7 +97750,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -97933,7 +97933,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -139085,7 +139085,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -139194,7 +139194,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -157591,7 +157591,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -157754,7 +157754,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -164344,7 +164344,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -164528,7 +164528,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -191493,7 +191493,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -192182,7 +192182,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -204323,7 +204323,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -205086,7 +205086,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",diff --git a/website/versioned_docs/version-3.5/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.5/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.5/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.5/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.5/examples/http_crawler.ts b/website/versioned_docs/version-3.5/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.5/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.5/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.5/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.5/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.5/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.5/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.5/experiments/request_locking.mdx b/website/versioned_docs/version-3.5/experiments/request_locking.mdxindex f4f00116f..f973f82e9 100644--- a/website/versioned_docs/version-3.5/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.5/experiments/request_locking.mdx@@ -67,8 +67,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.5/guides/docker_browser_js.txt b/website/versioned_docs/version-3.5/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.5/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.5/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.5/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.5/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.5/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.5/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.5/guides/docker_node_js.txt b/website/versioned_docs/version-3.5/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.5/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.5/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.5/guides/docker_node_ts.txt b/website/versioned_docs/version-3.5/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.5/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.5/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.5/guides/request_storage.mdx b/website/versioned_docs/version-3.5/guides/request_storage.mdxindex 2993b8371..3c2511853 100644--- a/website/versioned_docs/version-3.5/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.5/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.5/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.5/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.5/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.5/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.5/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.5/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.5/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.5/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.5/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.5/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.5/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.5/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.5/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.5/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.5/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.5/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.6/api-typedoc.json b/website/versioned_docs/version-3.6/api-typedoc.jsonindex 3e07e9a82..af59d0be8 100644--- a/website/versioned_docs/version-3.6/api-typedoc.json+++ b/website/versioned_docs/version-3.6/api-typedoc.json@@ -4935,7 +4935,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6157,7 +6157,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -14733,7 +14733,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -14994,7 +14994,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -15417,7 +15417,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -105820,7 +105820,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -106163,7 +106163,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -157761,7 +157761,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -158002,7 +158002,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -179396,7 +179396,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -179719,7 +179719,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -187473,7 +187473,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -187817,7 +187817,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -196291,7 +196291,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -196635,7 +196635,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -225161,7 +225161,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -226081,7 +226081,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -239768,7 +239768,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -240776,7 +240776,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -255381,7 +255381,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -255429,7 +255429,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.6/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.6/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.6/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.6/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.6/examples/http_crawler.ts b/website/versioned_docs/version-3.6/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.6/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.6/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.6/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.6/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.6/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.6/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.6/experiments/request_locking.mdx b/website/versioned_docs/version-3.6/experiments/request_locking.mdxindex f4f00116f..f973f82e9 100644--- a/website/versioned_docs/version-3.6/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.6/experiments/request_locking.mdx@@ -67,8 +67,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.6/guides/docker_browser_js.txt b/website/versioned_docs/version-3.6/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.6/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.6/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.6/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.6/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.6/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.6/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.6/guides/docker_node_js.txt b/website/versioned_docs/version-3.6/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.6/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.6/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.6/guides/docker_node_ts.txt b/website/versioned_docs/version-3.6/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.6/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.6/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.6/guides/request_storage.mdx b/website/versioned_docs/version-3.6/guides/request_storage.mdxindex 2993b8371..3c2511853 100644--- a/website/versioned_docs/version-3.6/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.6/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.6/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.6/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.6/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.6/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.6/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.6/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.6/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.6/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.6/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.6/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.6/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.6/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.6/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.6/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.6/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.6/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.7/api-typedoc.json b/website/versioned_docs/version-3.7/api-typedoc.jsonindex bdcd7e68f..da1c122a8 100644--- a/website/versioned_docs/version-3.7/api-typedoc.json+++ b/website/versioned_docs/version-3.7/api-typedoc.json@@ -5047,7 +5047,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6269,7 +6269,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -14985,7 +14985,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -15246,7 +15246,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -15669,7 +15669,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -106279,7 +106279,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -106622,7 +106622,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -158350,7 +158350,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -158591,7 +158591,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -180516,7 +180516,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -180839,7 +180839,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -188720,7 +188720,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -189064,7 +189064,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -197660,7 +197660,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -198004,7 +198004,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -226670,7 +226670,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -227590,7 +227590,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -241417,7 +241417,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -242425,7 +242425,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -254826,7 +254826,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.load('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.load('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -258988,7 +258988,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -259036,7 +259036,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.7/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.7/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.7/examples/http_crawler.ts b/website/versioned_docs/version-3.7/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.7/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.7/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.7/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.7/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.7/experiments/request_locking.mdx b/website/versioned_docs/version-3.7/experiments/request_locking.mdxindex f4f00116f..f973f82e9 100644--- a/website/versioned_docs/version-3.7/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.7/experiments/request_locking.mdx@@ -67,8 +67,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.7/guides/docker_browser_js.txt b/website/versioned_docs/version-3.7/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.7/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.7/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.7/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.7/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.7/guides/docker_node_js.txt b/website/versioned_docs/version-3.7/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.7/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.7/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.7/guides/docker_node_ts.txt b/website/versioned_docs/version-3.7/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.7/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.7/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.7/guides/request_storage.mdx b/website/versioned_docs/version-3.7/guides/request_storage.mdxindex 2993b8371..3c2511853 100644--- a/website/versioned_docs/version-3.7/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.7/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.7/introduction/01-setting-up.mdxindex a69852298..15e064863 100644--- a/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.7/introduction/01-setting-up.mdx@@ -49,10 +49,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdxindex 52f2a99f0..3fd95a59f 100644--- a/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.7/introduction/03-adding-urls.mdx@@ -70,7 +70,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.7/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.7/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.7/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.8/api-typedoc.json b/website/versioned_docs/version-3.8/api-typedoc.jsonindex 2642a98cd..8b38399b0 100644--- a/website/versioned_docs/version-3.8/api-typedoc.json+++ b/website/versioned_docs/version-3.8/api-typedoc.json@@ -5625,7 +5625,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6889,7 +6889,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -16190,7 +16190,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -16525,7 +16525,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -16951,7 +16951,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -108141,7 +108141,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -108558,7 +108558,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -138841,7 +138841,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -162948,7 +162948,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -163283,7 +163283,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -170974,7 +170974,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -186791,7 +186791,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -187188,7 +187188,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -195654,7 +195654,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -196072,7 +196072,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -205267,7 +205267,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -205685,7 +205685,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -254732,7 +254732,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -255726,7 +255726,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -270172,7 +270172,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -271254,7 +271254,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -283739,7 +283739,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.load('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.load('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -288320,7 +288320,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -288368,7 +288368,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.8/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.8/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.8/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.8/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.8/examples/http_crawler.ts b/website/versioned_docs/version-3.8/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.8/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.8/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.8/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.8/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.8/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.8/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.8/experiments/request_locking.mdx b/website/versioned_docs/version-3.8/experiments/request_locking.mdxindex f4f00116f..f973f82e9 100644--- a/website/versioned_docs/version-3.8/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.8/experiments/request_locking.mdx@@ -67,8 +67,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.8/guides/docker_browser_js.txt b/website/versioned_docs/version-3.8/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.8/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.8/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.8/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.8/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.8/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.8/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.8/guides/docker_node_js.txt b/website/versioned_docs/version-3.8/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.8/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.8/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.8/guides/docker_node_ts.txt b/website/versioned_docs/version-3.8/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.8/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.8/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.8/guides/request_storage.mdx b/website/versioned_docs/version-3.8/guides/request_storage.mdxindex 64c2efeb9..a48581ce2 100644--- a/website/versioned_docs/version-3.8/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.8/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.8/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.8/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.8/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.8/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.8/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.8/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.8/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.8/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.8/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.8/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.8/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.8/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.8/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.8/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.8/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.8/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/versioned_docs/version-3.9/api-typedoc.json b/website/versioned_docs/version-3.9/api-typedoc.jsonindex e88176bd7..6ca3f7f6d 100644--- a/website/versioned_docs/version-3.9/api-typedoc.json+++ b/website/versioned_docs/version-3.9/api-typedoc.json@@ -5677,7 +5677,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -5722,7 +5722,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -6986,7 +6986,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -16435,7 +16435,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -16480,7 +16480,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -16815,7 +16815,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -17248,7 +17248,7 @@},{"kind": "text",- "text": "\n\nFor more details, check out the [example](https://crawlee.dev/docs/examples/playwright-crawler-firefox)."+ "text": "\n\nFor more details, check out the [example](https://crawlee.dev/js/docs/examples/playwright-crawler-firefox)."}]}@@ -109088,7 +109088,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -109505,7 +109505,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -164809,7 +164809,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -164854,7 +164854,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -165189,7 +165189,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -173034,7 +173034,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -189392,7 +189392,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -189789,7 +189789,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -198365,7 +198365,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -198783,7 +198783,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -208229,7 +208229,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -208647,7 +208647,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -240129,7 +240129,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -258639,7 +258639,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -258684,7 +258684,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -259678,7 +259678,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -274411,7 +274411,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -274456,7 +274456,7 @@},{"kind": "text",- "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"+ "text": " objects.\n\nCheck out the [Crawl a website with relative links](https://crawlee.dev/js/docs/examples/crawl-relative-links) example\nfor more details regarding its usage.\n\n**Example usage**\n\n"},{"kind": "code",@@ -275538,7 +275538,7 @@},{"kind": "text",- "text": "](https://crawlee.dev/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"+ "text": "](https://crawlee.dev/js/docs/guides/got-scraping), allowing to override the request\noptions on the fly.\n\nThis is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests).\nCheck the [Skipping navigations for certain requests](https://crawlee.dev/js/docs/examples/skip-navigation) example for\nmore detailed explanation of how to do that.\n\n"},{"kind": "code",@@ -286742,7 +286742,7 @@},{"kind": "code",- "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"+ "text": "```javascript\n// Load the robots.txt file\nconst robots = await RobotsFile.find('https://crawlee.dev/js/docs/introduction/first-crawler');\n\n// Check if a URL should be crawled according to robots.txt\nconst url = 'https://crawlee.dev/js/api/puppeteer-crawler/class/PuppeteerCrawler';\nif (robots.isAllowed(url)) {\n await crawler.addRequests([url]);\n}\n\n// Enqueue all links in the sitemap(s)\nawait crawler.addRequests(await robots.parseUrlsFromSitemaps());\n```"}]},@@ -291638,7 +291638,7 @@},{"kind": "text",- "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"+ "text": "](https://www.npmjs.com/package/crawlee) NPM package.\n\n> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev)** 👈\n\n## Installation\n\nWe recommend visiting the [Introduction tutorial](https://crawlee.dev/js/docs/introduction) in Crawlee documentation for more information.\n\n> Crawlee requires **Node.js 16 or higher**.\n\n### With Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.\n\n"},{"kind": "code",@@ -291686,7 +291686,7 @@},{"kind": "text",- "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/docs/guides/configuration), [Request storage](https://crawlee.dev/docs/guides/request-storage) and [Result storage](https://crawlee.dev/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."+ "text": " in the current working directory. You can override this directory via Crawlee configuration. For details, see [Configuration guide](https://crawlee.dev/js/docs/guides/configuration), [Request storage](https://crawlee.dev/js/docs/guides/request-storage) and [Result storage](https://crawlee.dev/js/docs/guides/result-storage).\n\n## 🛠 Features\n\n- Single interface for **HTTP and headless browser** crawling\n- Persistent **queue** for URLs to crawl (breadth & depth first)\n- Pluggable **storage** of both tabular data and files\n- Automatic **scaling** with available system resources\n- Integrated **proxy rotation** and session management\n- Lifecycles customizable with **hooks**\n- **CLI** to bootstrap your projects\n- Configurable **routing**, **error handling** and **retries**\n- **Dockerfiles** ready to deploy\n- Written in **TypeScript** with generics\n\n### 👾 HTTP crawling\n\n- Zero config **HTTP2 support**, even for proxies\n- Automatic generation of **browser-like headers**\n- Replication of browser **TLS fingerprints**\n- Integrated fast **HTML parsers**. Cheerio and JSDOM\n- Yes, you can scrape **JSON APIs** as well\n\n### 💻 Real browser crawling\n\n- JavaScript **rendering** and **screenshots**\n- **Headless** and **headful** support\n- Zero-config generation of **human-like fingerprints**\n- Automatic **browser management**\n- Use **Playwright** and **Puppeteer** with the same interface\n- **Chrome**, **Firefox**, **Webkit** and many others\n\n## Usage on the Apify platform\n\nCrawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://sdk.apify.com) to learn more about deploying Crawlee to the Apify platform.\n\n## Support\n\nIf you find any bug or issue with Crawlee, please [submit an issue on GitHub](https://github.com/apify/crawlee/issues). For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).\n\n## Contributing\n\nYour code contributions are welcome, and you'll be praised to eternity! If you have any ideas for improvements, either submit an issue or create a pull request. For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) file for details."}],"symbolIdMap": {diff --git a/website/versioned_docs/version-3.9/examples/cheerio_crawler.ts b/website/versioned_docs/version-3.9/examples/cheerio_crawler.tsindex c496c1572..8d63a9e05 100644--- a/website/versioned_docs/version-3.9/examples/cheerio_crawler.ts+++ b/website/versioned_docs/version-3.9/examples/cheerio_crawler.ts@@ -25,7 +25,7 @@ const crawler = new CheerioCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/cheerio-crawler/interface/CheerioCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - $: the cheerio object containing parsed HTMLdiff --git a/website/versioned_docs/version-3.9/examples/http_crawler.ts b/website/versioned_docs/version-3.9/examples/http_crawler.tsindex 4052df131..52c9104b8 100644--- a/website/versioned_docs/version-3.9/examples/http_crawler.ts+++ b/website/versioned_docs/version-3.9/examples/http_crawler.ts@@ -25,7 +25,7 @@ const crawler = new HttpCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/http-crawler/interface/HttpCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/http-crawler/interface/HttpCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - body: the HTML code of the current pagediff --git a/website/versioned_docs/version-3.9/examples/jsdom_crawler.ts b/website/versioned_docs/version-3.9/examples/jsdom_crawler.tsindex e72034113..be5cc9833 100644--- a/website/versioned_docs/version-3.9/examples/jsdom_crawler.ts+++ b/website/versioned_docs/version-3.9/examples/jsdom_crawler.ts@@ -25,7 +25,7 @@ const crawler = new JSDOMCrawler({// This function will be called for each URL to crawl.// It accepts a single parameter, which is an object with options as:- // https://crawlee.dev/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler+ // https://crawlee.dev/js/api/jsdom-crawler/interface/JSDOMCrawlerOptions#requestHandler// We use for demonstration only 2 of them:// - request: an instance of the Request class with information such as the URL that is being crawled and HTTP method// - window: the JSDOM window objectdiff --git a/website/versioned_docs/version-3.9/experiments/request_locking.mdx b/website/versioned_docs/version-3.9/experiments/request_locking.mdxindex f4f00116f..f973f82e9 100644--- a/website/versioned_docs/version-3.9/experiments/request_locking.mdx+++ b/website/versioned_docs/version-3.9/experiments/request_locking.mdx@@ -67,8 +67,8 @@ import { RequestQueueV2 } from 'crawlee';const queue = await RequestQueueV2.open('my-locking-queue');await queue.addRequests([{ url: 'https://crawlee.dev' },- { url: 'https://crawlee.dev/docs' },- { url: 'https://crawlee.dev/api' },+ { url: 'https://crawlee.dev/js/docs' },+ { url: 'https://crawlee.dev/js/api' },]);```diff --git a/website/versioned_docs/version-3.9/guides/docker_browser_js.txt b/website/versioned_docs/version-3.9/guides/docker_browser_js.txtindex afe996891..3eeaca56d 100644--- a/website/versioned_docs/version-3.9/guides/docker_browser_js.txt+++ b/website/versioned_docs/version-3.9/guides/docker_browser_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16diff --git a/website/versioned_docs/version-3.9/guides/docker_browser_ts.txt b/website/versioned_docs/version-3.9/guides/docker_browser_ts.txtindex e6477ffc3..1d8606497 100644--- a/website/versioned_docs/version-3.9/guides/docker_browser_ts.txt+++ b/website/versioned_docs/version-3.9/guides/docker_browser_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node-playwright-chrome:16 AS builderdiff --git a/website/versioned_docs/version-3.9/guides/docker_node_js.txt b/website/versioned_docs/version-3.9/guides/docker_node_js.txtindex 4018d6b99..3b8961109 100644--- a/website/versioned_docs/version-3.9/guides/docker_node_js.txt+++ b/website/versioned_docs/version-3.9/guides/docker_node_js.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16diff --git a/website/versioned_docs/version-3.9/guides/docker_node_ts.txt b/website/versioned_docs/version-3.9/guides/docker_node_ts.txtindex 43815f97d..ca3841cbf 100644--- a/website/versioned_docs/version-3.9/guides/docker_node_ts.txt+++ b/website/versioned_docs/version-3.9/guides/docker_node_ts.txt@@ -1,5 +1,5 @@# Specify the base Docker image. You can read more about-# the available images at https://crawlee.dev/docs/guides/docker-images+# the available images at https://crawlee.dev/js/docs/guides/docker-images# You can also use any other image from Docker Hub.FROM apify/actor-node:16 AS builderdiff --git a/website/versioned_docs/version-3.9/guides/request_storage.mdx b/website/versioned_docs/version-3.9/guides/request_storage.mdxindex 64c2efeb9..a48581ce2 100644--- a/website/versioned_docs/version-3.9/guides/request_storage.mdx+++ b/website/versioned_docs/version-3.9/guides/request_storage.mdx@@ -65,7 +65,7 @@ The following code demonstrates the usage of the request queue:</TabItem></Tabs>-To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/docs/examples/puppeteer-crawler) example.+To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.## Request listdiff --git a/website/versioned_docs/version-3.9/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.9/introduction/01-setting-up.mdxindex b1b131dab..c4dbddd29 100644--- a/website/versioned_docs/version-3.9/introduction/01-setting-up.mdx+++ b/website/versioned_docs/version-3.9/introduction/01-setting-up.mdx@@ -47,10 +47,10 @@ You will see log messages in the terminal as Crawlee boots up and starts scrapin```logINFO PlaywrightCrawler: Starting the crawlINFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'-INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'+INFO PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'```You can always terminate the crawl with a keypress in the terminal:diff --git a/website/versioned_docs/version-3.9/introduction/03-adding-urls.mdx b/website/versioned_docs/version-3.9/introduction/03-adding-urls.mdxindex 21e6e58b0..387b86fb9 100644--- a/website/versioned_docs/version-3.9/introduction/03-adding-urls.mdx+++ b/website/versioned_docs/version-3.9/introduction/03-adding-urls.mdx@@ -69,7 +69,7 @@ This means that no new requests will be started after the 20th request is finishThere are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:```html-<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>+<a href="https://crawlee.dev/js/docs/introduction">This is a link to Crawlee introduction</a>```Since this is the most common case, it is also the `enqueueLinks` default.diff --git a/website/versioned_docs/version-3.9/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.9/quick-start/quick_start_cheerio.txtindex e68fa4133..220c40a19 100644--- a/website/versioned_docs/version-3.9/quick-start/quick_start_cheerio.txt+++ b/website/versioned_docs/version-3.9/quick-start/quick_start_cheerio.txt@@ -1,5 +1,5 @@INFO CheerioCrawler: Starting the crawlINFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'-INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'+INFO CheerioCrawler: Title of https://crawlee.dev/js/docs/guides is 'Guides | Crawlee'diff --git a/website/versioned_docs/version-3.9/upgrading/upgrading_v3.md b/website/versioned_docs/version-3.9/upgrading/upgrading_v3.mdindex bd7f43018..84f331f53 100644--- a/website/versioned_docs/version-3.9/upgrading/upgrading_v3.md+++ b/website/versioned_docs/version-3.9/upgrading/upgrading_v3.md@@ -20,18 +20,18 @@ Up until version 3 of `apify`, the package contained both scraping related toolsThe [`crawlee`](https://www.npmjs.com/package/crawlee) package consists of several smaller packages, released separately under `@crawlee` namespace:-- [`@crawlee/core`](https://crawlee.dev/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes-- [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler): exports `CheerioCrawler`-- [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler): exports `PlaywrightCrawler`-- [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler): exports `PuppeteerCrawler`-- [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler): exports `JSDOMCrawler`-- [`@crawlee/basic`](https://crawlee.dev/api/basic-crawler): exports `BasicCrawler`-- [`@crawlee/http`](https://crawlee.dev/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/api/cheerio-crawler))-- [`@crawlee/browser`](https://crawlee.dev/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/api/puppeteer-crawler))-- [`@crawlee/memory-storage`](https://crawlee.dev/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative-- [`@crawlee/browser-pool`](https://crawlee.dev/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package-- [`@crawlee/utils`](https://crawlee.dev/api/utils): utility methods-- [`@crawlee/types`](https://crawlee.dev/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/api/core/interface/StorageClient)+- [`@crawlee/core`](https://crawlee.dev/js/api/core): the base for all the crawler implementations, also contains things like `Request`, `RequestQueue`, `RequestList` or `Dataset` classes+- [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler): exports `CheerioCrawler`+- [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler): exports `PlaywrightCrawler`+- [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler): exports `PuppeteerCrawler`+- [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler): exports `JSDOMCrawler`+- [`@crawlee/basic`](https://crawlee.dev/js/api/basic-crawler): exports `BasicCrawler`+- [`@crawlee/http`](https://crawlee.dev/js/api/http-crawler): exports `HttpCrawler` (which is used for creating [`@crawlee/jsdom`](https://crawlee.dev/js/api/jsdom-crawler) and [`@crawlee/cheerio`](https://crawlee.dev/js/api/cheerio-crawler))+- [`@crawlee/browser`](https://crawlee.dev/js/api/browser-crawler): exports `BrowserCrawler` (which is used for creating [`@crawlee/playwright`](https://crawlee.dev/js/api/playwright-crawler) and [`@crawlee/puppeteer`](https://crawlee.dev/js/api/puppeteer-crawler))+- [`@crawlee/memory-storage`](https://crawlee.dev/js/api/memory-storage): [`@apify/storage-local`](https://npmjs.com/package/@apify/storage-local) alternative+- [`@crawlee/browser-pool`](https://crawlee.dev/js/api/browser-pool): previously [`browser-pool`](https://npmjs.com/package/browser-pool) package+- [`@crawlee/utils`](https://crawlee.dev/js/api/utils): utility methods+- [`@crawlee/types`](https://crawlee.dev/js/api/types): holds TS interfaces mainly about the [`StorageClient`](https://crawlee.dev/js/api/core/interface/StorageClient)### Installing Crawleediff --git a/website/yarn.lock b/website/yarn.lockindex ab60e1951..fff038c45 100644--- a/website/yarn.lock+++ b/website/yarn.lock@@ -8279,6 +8279,13 @@ __metadata:languageName: nodelinkType: hard+"github-buttons@npm:^2.22.0":+ version: 2.29.1+ resolution: "github-buttons@npm:2.29.1"+ checksum: 10c0/f403cce395b09acab1bc3c2d08549d1130f77a56338d7c7be84cd101774eeb681b0f9a050c6e263dc585c60f3a3cf406be762afe2400449ad842e41732a3f897+ languageName: node+ linkType: hard+"github-slugger@npm:^1.5.0":version: 1.5.0resolution: "github-slugger@npm:1.5.0"@@ -13470,6 +13477,17 @@ __metadata:languageName: nodelinkType: hard+"react-github-btn@npm:^1.4.0":+ version: 1.4.0+ resolution: "react-github-btn@npm:1.4.0"+ dependencies:+ github-buttons: "npm:^2.22.0"+ peerDependencies:+ react: ">=16.3.0"+ checksum: 10c0/ce21b1081e8985fce93f19deb3ab3e20b36e80508fcd9346eb617728b2e242bea845434296a9b8acbc339259551d58e43d840dbe9ec03c21a7094a7a9038986e+ languageName: node+ linkType: hard+"react-helmet-async@npm:@slorber/react-helmet-async@*, react-helmet-async@npm:@slorber/[email protected]":version: 1.3.0resolution: "@slorber/react-helmet-async@npm:1.3.0"@@ -14206,6 +14224,7 @@ __metadata:raw-loader: "npm:^4.0.2"react: "npm:^19.0.0"react-dom: "npm:^19.0.0"+ react-github-btn: "npm:^1.4.0"react-lite-youtube-embed: "npm:^2.3.52"rimraf: "npm:^6.0.0"stream-browserify: "npm:^3.0.0"