Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Changing www.apify.com to apify.com
Marek Trunkat6e9f4eca6709
diff --git a/README.md b/README.mdindex 1a7bf453e..3158e4b4c 100644--- a/README.md+++ b/README.md@@ -10,8 +10,8 @@ to maintain queues of URLs to crawl, store crawling results to a local filesysterotate proxies and much more.The SDK is available as the <a href="https://www.npmjs.com/package/apify" target="_blank"><code>apify</code></a> NPM package.It can be used either stand-alone in your own applications-or in <a href="https://www.apify.com/docs/actor" target="_blank">actors</a>-running on the <a href="https://www.apify.com/" target="_blank">Apify Cloud</a>.+or in <a href="https://apify.com/docs/actor" target="_blank">actors</a>+running on the <a href="https://apify.com/" target="_blank">Apify Cloud</a>.**View full documentation, guides and examples on the dedicated <a href="https://sdk.apify.com" target="_blank">Apify SDK project website</a>**@@ -77,7 +77,7 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"></li><li><a href="https://sdk.apify.com/docs/api/requestqueue"><code>RequestQueue</code></a>- - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://www.apify.com" target="_blank">Apify Cloud</a>.+ - Represents a queue of URLs to crawl, which is stored either on a local filesystem or in the <a href="https://apify.com" target="_blank">Apify Cloud</a>.The queue is used for deep crawling of websites, where you start withseveral URLs and then recursively follow links to other pages.The data structure supports both breadth-first and depth-first crawling orders.@@ -128,7 +128,7 @@ npm install apify --save```Run the following example to perform a recursive crawl of a website using Puppeteer.-For more examples showcasing various features of the Apify SDK,+For more examples showcasing various features of the Apify SDK,[see the Examples section of the documentation](https://sdk.apify.com/docs/examples/basiccrawler).```javascript@@ -170,7 +170,7 @@ and [Data storage](https://sdk.apify.com/docs/guides/datastorage).To avoid the need to set the environment variables manually,to create a boilerplate of your project,and to enable pushing and running your code on the-<a href="https://www.apify.com" target="_blank">Apify Cloud</a>,+<a href="https://apify.com" target="_blank">Apify Cloud</a>,you can use the<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a>(CLI) tool.@@ -213,23 +213,23 @@ apify pushYour script will be uploaded to the Apify Cloud and built there so that it can be run.For more information, view the-<a href="https://www.apify.com/docs/cli" target="_blank">Apify CLI</a>+<a href="https://apify.com/docs/cli" target="_blank">Apify CLI</a>and-<a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a>+<a href="https://apify.com/docs/actor" target="_blank">Apify Actor</a>documentation.### Usage on the Apify CloudYou can also develop your web scraping projectin an online code editor directly on the-<a href="https://www.apify.com" target="_blank">Apify Cloud</a>.+<a href="https://apify.com" target="_blank">Apify Cloud</a>.You'll need to have an Apify Account.Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>,page in the app, click <i>Create new</i> and then go to the<i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.For more information, view the-<a href="https://www.apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.+<a href="https://apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.## Supportdiff --git a/docs/guides/data_storage.md b/docs/guides/data_storage.mdindex b1c12201f..9854e4ad2 100644--- a/docs/guides/data_storage.md+++ b/docs/guides/data_storage.md@@ -5,7 +5,7 @@ title: Data StorageThe Apify SDK has several data storage types that are useful for specific tasks.The data is stored either on local disk to a directory defined by the `APIFY_LOCAL_STORAGE_DIR` environment variable,-or on the <a href="https://www.apify.com" target="_blank">Apify Cloud</a> under the user account identified by the API token defined by the `APIFY_TOKEN` environment variable.+or on the <a href="https://apify.com" target="_blank">Apify Cloud</a> under the user account identified by the API token defined by the `APIFY_TOKEN` environment variable.If neither of these variables is defined, by default Apify SDK sets `APIFY_LOCAL_STORAGE_DIR`to `./apify_storage` in the current working directory and prints a warning.@@ -16,9 +16,9 @@ No code changes are needed.**Related links**-* <a href="https://www.apify.com/docs/storage" target="_blank">Apify cloud storage documentation</a>+* <a href="https://apify.com/docs/storage" target="_blank">Apify cloud storage documentation</a>* <a href="https://my.apify.com/storage" target="_blank">View storage in Apify app</a>-* <a href="https://www.apify.com/docs/api/v2#/reference/key-value-stores" target="_blank">API reference</a>+* <a href="https://apify.com/docs/api/v2#/reference/key-value-stores" target="_blank">API reference</a>## Key-value store@@ -86,11 +86,11 @@ You can imagine a dataset as a table, where each object is a row and its attribuDataset is an append-only storage - you can only add new records to it but you cannot modify or removeexisting records.-When the dataset is stored in the <a href="https://www.apify.com" target="_blank">Apify Cloud</a>,+When the dataset is stored in the <a href="https://apify.com" target="_blank">Apify Cloud</a>,you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS.The datasets are displayed on the actor run details page and in the <a href="https://my.apify.com/storage" target="_blank">Storage</a>section in the Apify app. The actual data is exported using the-<a href="https://www.apify.com/docs/api/v2#/reference/datasets/item-collection/get-items" target="_blank">Get dataset items</a> Apify API endpoint.+<a href="https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items" target="_blank">Get dataset items</a> Apify API endpoint.This way you can easily share crawling results.Each actor run is associated with a **default dataset**, which is created exclusively for the actor run.diff --git a/docs/guides/environment_variables.md b/docs/guides/environment_variables.mdindex 0035bf789..b5f3130d4 100644--- a/docs/guides/environment_variables.md+++ b/docs/guides/environment_variables.md@@ -36,7 +36,7 @@ The following table shows the basic environment variables used by Apify SDK:<tr><td><code>APIFY_PROXY_PASSWORD</code></td><td>- Optional password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address rotation.+ Optional password to <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address rotation.If you have have an Apify Account, you can find the password on the<a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.This feature is optional. You can use your own proxies or no proxies at all.diff --git a/docs/guides/getting_started.md b/docs/guides/getting_started.mdindex 4dad3b556..d9f09dce1 100644--- a/docs/guides/getting_started.md+++ b/docs/guides/getting_started.md@@ -76,7 +76,7 @@ Did you see all that? If you did, congratulations! You're ready to go!## Setting up on the Apify PlatformMaybe you don't have Node.js installed and don't want the hassle. Or you can't install anything on your computer because you're using a company provided one. Or perhaps you'd just prefer to start working in the cloud right away. Well, no worries, we've got you covered.-The <a href="https://my.apify.com" target="_blank">Apify Platform</a> is the foundational product of <a href="https://www.apify.com" target="_blank">Apify</a>. It's a serverless cloud computing platform, specifically designed for any web automation jobs, that may include crawling and scraping, but really works amazing for any batch jobs and long running tasks.+The <a href="https://my.apify.com" target="_blank">Apify Platform</a> is the foundational product of <a href="https://apify.com" target="_blank">Apify</a>. It's a serverless cloud computing platform, specifically designed for any web automation jobs, that may include crawling and scraping, but really works amazing for any batch jobs and long running tasks.It comes with a free account, so let's go to our<a href="https://my.apify.com/sign-up" target="_blank">sign-up page</a>@@ -90,7 +90,7 @@ In the page that shows after clicking on Actors in the left menu, choose **CreatNow click on the **Sources** tab at the top. Disregard the version and environment variables inputs for now and proceed directly to **Source code**. This is where you develop the actor, if you choose not to do it locally. Just press **Run** below the **Source code** panel. It will automatically build and run the example source code. You should start seeing log messages that represent the build and after the build is complete, log messages of the running actor. Feel free to check out the other **Run** tabs, such as **Info**, where you can find useful information about the run, or **Key-value-store**, where the actor's **INPUT** and **OUTPUT** are stored.Good job. You're now ready to run your own source code on the Apify Platform. For more information visit the-<a href="https://www.apify.com/docs/actor" target="_blank">Actor documentation page</a>,+<a href="https://apify.com/docs/actor" target="_blank">Actor documentation page</a>,where you'll find everything about the platform's various options.## First crawler@@ -132,7 +132,7 @@ Let's start super easy. Visit one page, get its title and close. First of all weconst Apify = require('apify');```-Easy right? It doesn't get much more difficult, trust me. For the purposes of this tutorial, we'll be scraping our own webpage <a href="https://www.apify.com" target="_blank">https://www.apify.com</a>. Now, to get there, we need a `Request` with the page's URL in one of our sources, `RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.+Easy right? It doesn't get much more difficult, trust me. For the purposes of this tutorial, we'll be scraping our own webpage <a href="https://apify.com" target="_blank">https://apify.com</a>. Now, to get there, we need a `Request` with the page's URL in one of our sources, `RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.```jsconst Apify = require('apify');@@ -142,12 +142,12 @@ Apify.main(async () => {// First we create the request queue instance.const requestQueue = await Apify.openRequestQueue();// And then we add a request to it.- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });});```> If you're not familiar with the `async` and `await` keywords used in the example, trust that it is a native syntax in modern JavaScript and you can [learn more about it here](https://nikgrozev.com/2017/10/01/async-await/).-The [`requestQueue.addRequest()`](../api/requestqueue#RequestQueue+addRequest) function automatically converts the plain object we passed to it to a `Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://www.apify.com`. Now we need the `handlePageFunction`.+The [`requestQueue.addRequest()`](../api/requestqueue#RequestQueue+addRequest) function automatically converts the plain object we passed to it to a `Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://apify.com`. Now we need the `handlePageFunction`.```js// We'll define the function separately so it's more obvious.@@ -167,7 +167,7 @@ const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {const title = $('title').text();@@ -185,13 +185,13 @@ Apify.main(async () => {});```-And we're done! You just created your first crawler from scratch. It will download the HTML of `https://www.apify.com`, find the `<title>` element, get its text content and print it to console. Good job!+And we're done! You just created your first crawler from scratch. It will download the HTML of `https://apify.com`, find the `<title>` element, get its text content and print it to console. Good job!To run the code locally, copy and paste the code, if you haven't already typed it in yourself, to the `main.js` file in the `my-new-project` we created earlier and run `apify run` from that project's directory.To run the code on Apify Platform, just replace the original example with your new code and hit Run.-Whichever environment you choose, you should see the message `The title of "https://www.apify.com" is: Web Scraping, Data Extraction and Automation - Apify.` printed to the screen. If you do, congratulations and let's move onto some bigger challenges! And if you feel like you don't really know what just happened there, no worries, it will all become clear when you learn more about the `CheerioCrawler`.+Whichever environment you choose, you should see the message `The title of "https://apify.com" is: Web Scraping, Data Extraction and Automation - Apify.` printed to the screen. If you do, congratulations and let's move onto some bigger challenges! And if you feel like you don't really know what just happened there, no worries, it will all become clear when you learn more about the `CheerioCrawler`.## CheerioCrawler aka jQuery crawler@@ -247,7 +247,7 @@ const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {const title = $('title').text();@@ -274,7 +274,7 @@ Earlier we said that we would let the crawler:So let's get to it!#### Finding new links-There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute. For example `<a href="https://www.apify.com/library>This is a link to Apify Library</a>`. To do this, we need to update our Cheerio function.+There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute. For example `<a href="https://apify.com/library>This is a link to Apify Library</a>`. To do this, we need to update our Cheerio function.```jsconst links = $('a[href]').map((i, el) => $(el).attr('href')).get();@@ -354,7 +354,7 @@ const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {const title = $('title').text();@@ -525,7 +525,7 @@ const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {const title = $('title').text();@@ -562,7 +562,7 @@ const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: 'https://www.apify.com' });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {const title = $('title').text();@@ -597,7 +597,7 @@ And that's it! No more parsing the links from HTML using Cheerio, filtering themWe hear you young padawan! First, learn how to crawl, you must. Only then, save data, you can!### Making a library crawler-Fortunately, we don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape structured data. The <a href="https://www.apify.com/library" target="_blank">Apify Library</a> is a library of public actors that anyone can grab and use. You can find ready-made solutions for crawling Google Places, Amazon, Google SERPs, Booking, Kickstarter and many other websites. Feel free to check them out! It also poses a great place to practice our jedi scraping skills since it has categories, lists and details. That's almost like our imaginary `online-store.com` from the previous chapter.+Fortunately, we don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape structured data. The <a href="https://apify.com/library" target="_blank">Apify Library</a> is a library of public actors that anyone can grab and use. You can find ready-made solutions for crawling Google Places, Amazon, Google SERPs, Booking, Kickstarter and many other websites. Feel free to check them out! It also poses a great place to practice our jedi scraping skills since it has categories, lists and details. That's almost like our imaginary `online-store.com` from the previous chapter.### The importance of having a planSometimes scraping is really straightforward, but most of the times, it really pays out to do a little bit of research first. How is the website structured? Can I scrape it only with HTTP requests (read "with `CheerioCrawler`") or would I need a full browser solution? Are there any anti-scraping protections in place? Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API. Jakub, one of Apify's founders wrote a <a href="https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c" target="_blank">great article about all the different techniques</a> and tips and tricks so make sure to check that out!@@ -626,7 +626,7 @@ This is where we start our crawl. It's convenient to start as close to our dataOnce we look at the `www.apify.com/library` page more carefully though, we see that the categories themselves produce URLs that we can use to access those individual categories.```-https://www.apify.com/library?category=ENTERTAINMENT+https://apify.com/library?category=ENTERTAINMENT```Should we write down all the category URLs down and use all of them as start URLs? It's definitely possible, but what if a new category appears on the page later? We would not learn about it unless we manually visit the page and inspect it again. So scraping the category links off the library page definitely makes sense. This way we always get an up to date list of categories.@@ -640,11 +640,11 @@ So we're back to the pre-selected list of URLs. Since we cannot scrape the listTherefore, after careful consideration, we've determined that we should use multiple start URLs and that they should look as follows:```-https://www.apify.com/library?type=acts&category=TRAVEL-https://www.apify.com/library?type=acts&category=ECOMMERCE-https://www.apify.com/library?type=acts&category=ENTERTAINMENT+https://apify.com/library?type=acts&category=TRAVEL+https://apify.com/library?type=acts&category=ECOMMERCE+https://apify.com/library?type=acts&category=ENTERTAINMENT```-> The `type=acts` query parameter comes from selecting `Actors only` in the `Show` dropdown. This is in line with us only wanting to scrape actors' data. If you're wondering how we've created these URLs, simply visit the `https://www.apify.com/library` page, select `Actors only` in the `Show` dropdown and click on one of the categories in the left hand menu. The correct URL will show up in your browser's address bar.+> The `type=acts` query parameter comes from selecting `Actors only` in the `Show` dropdown. This is in line with us only wanting to scrape actors' data. If you're wondering how we've created these URLs, simply visit the `https://apify.com/library` page, select `Actors only` in the `Show` dropdown and click on one of the categories in the left hand menu. The correct URL will show up in your browser's address bar.### The crawling strategyNow that we know where to start, we need to figure out where to go next. Since we've eliminated one level of crawling by selecting the categories manually, we only need to crawl the actor detail pages now. The algorithm therefore follows:@@ -663,9 +663,9 @@ Now that we know where to start, we need to figure out where to go next. Since w```jsconst sources = [- 'https://www.apify.com/library?type=acts&category=TRAVEL',- 'https://www.apify.com/library?type=acts&category=ECOMMERCE',- 'https://www.apify.com/library?type=acts&category=ENTERTAINMENT'+ 'https://apify.com/library?type=acts&category=TRAVEL',+ 'https://apify.com/library?type=acts&category=ECOMMERCE',+ 'https://apify.com/library?type=acts&category=ENTERTAINMENT'];const requestList = await Apify.openRequestList('categories', sources);@@ -702,9 +702,9 @@ const Apify = require('apify');Apify.main(async () => {const sources = [- 'https://www.apify.com/library?type=acts&category=TRAVEL',- 'https://www.apify.com/library?type=acts&category=ECOMMERCE',- 'https://www.apify.com/library?type=acts&category=ENTERTAINMENT'+ 'https://apify.com/library?type=acts&category=TRAVEL',+ 'https://apify.com/library?type=acts&category=ECOMMERCE',+ 'https://apify.com/library?type=acts&category=ENTERTAINMENT'];const requestList = await Apify.openRequestList('x', sources);@@ -734,7 +734,7 @@ You might be wondering how we got that `.item` selector. After analyzing the catAt time of this writing, there are only 2 actors in the Travel category, so we'll use this one for our examples, since it will make everything much less cluttered. Now, go to```-https://www.apify.com/library?type=acts&category=TRAVEL+https://apify.com/library?type=acts&category=TRAVEL```and open DevTools either by right clicking anywhere in the page and selecting `Inspect`, or by pressing `F12` or by any other means relevant to your system. Once you're there, you'll see a bunch of DevToolsy stuff and a view of the category page with the individual actor cards.@@ -811,7 +811,7 @@ Earlier we've learned that `pseudoUrls` are not required and if omitted, all linThis is probably the weirdest and most complicated addition to the list. This is not the place to talk at length about <a href="https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls" target="_blank">absolute and relative paths</a>, but in short, the links we encounter in a page can either be absolute, such as:```-https://www.apify.com/john-doe/my-actor+https://apify.com/john-doe/my-actor```or relative:@@ -838,9 +838,9 @@ const Apify = require('apify');Apify.main(async () => {const sources = [- 'https://www.apify.com/library?type=acts&category=TRAVEL',- 'https://www.apify.com/library?type=acts&category=ECOMMERCE',- 'https://www.apify.com/library?type=acts&category=ENTERTAINMENT'+ 'https://apify.com/library?type=acts&category=TRAVEL',+ 'https://apify.com/library?type=acts&category=ECOMMERCE',+ 'https://apify.com/library?type=acts&category=ENTERTAINMENT'];const requestList = await Apify.openRequestList('x', sources);@@ -892,7 +892,7 @@ At the beginning of this chapter, we've created a list of information we wantedSome information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!```js-// request.url = https://www.apify.com/johndoe/amazing-web-crawler+// request.url = https://apify.com/johndoe/amazing-web-crawlerconst urlArr = request.url.split('/').slice(-2); // ['johndoe', 'amazing-web-crawler']const uniqueIdentifier = urlArr.join('/'); // 'johndoe/amazing-web-crawler'@@ -997,9 +997,9 @@ const Apify = require('apify');Apify.main(async () => {const sources = [- 'https://www.apify.com/library?type=acts&category=TRAVEL',- 'https://www.apify.com/library?type=acts&category=ECOMMERCE',- 'https://www.apify.com/library?type=acts&category=ENTERTAINMENT'+ 'https://apify.com/library?type=acts&category=TRAVEL',+ 'https://apify.com/library?type=acts&category=ECOMMERCE',+ 'https://apify.com/library?type=acts&category=ENTERTAINMENT'];const requestList = await Apify.openRequestList('x', sources);@@ -1070,9 +1070,9 @@ const Apify = require('apify');Apify.main(async () => {const sources = [- 'https://www.apify.com/library?type=acts&category=TRAVEL',- 'https://www.apify.com/library?type=acts&category=ECOMMERCE',- 'https://www.apify.com/library?type=acts&category=ENTERTAINMENT'+ 'https://apify.com/library?type=acts&category=TRAVEL',+ 'https://apify.com/library?type=acts&category=ECOMMERCE',+ 'https://apify.com/library?type=acts&category=ENTERTAINMENT'];const requestList = await Apify.openRequestList('x', sources);@@ -1188,7 +1188,7 @@ Once we have that, we can load it in the actor and populate the crawler's sourceconst input = await Apify.getValue('INPUT');const sources = input.map(category => ({- url: `https://www.apify.com/library?type=acts&category=${category}`,+ url: `https://apify.com/library?type=acts&category=${category}`,userData: {label: 'CATEGORY'}@@ -1243,7 +1243,7 @@ async function getSources() {const input = await Apify.getValue('INPUT');return input.map(category => ({- url: `https://www.apify.com/library?type=acts&category=${category}`,+ url: `https://apify.com/library?type=acts&category=${category}`,userData: {label: 'CATEGORY'}diff --git a/docs/guides/getting_started_quick.md b/docs/guides/getting_started_quick.mdindex 85277fe5f..dc064f1be 100644--- a/docs/guides/getting_started_quick.md+++ b/docs/guides/getting_started_quick.md@@ -55,7 +55,7 @@ and [Data storage](datastorage).To avoid the need to set the environment variables manually,to create a boilerplate of your project,and to enable pushing and running your code on the-<a href="https://www.apify.com" target="_blank">Apify Cloud</a>,+<a href="https://apify.com" target="_blank">Apify Cloud</a>,you can use the<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a>(CLI) tool.@@ -98,20 +98,20 @@ apify pushYour script will be uploaded to the Apify Cloud and built there so that it can be run.For more information, view the-<a href="https://www.apify.com/docs/cli" target="_blank">Apify CLI</a>+<a href="https://apify.com/docs/cli" target="_blank">Apify CLI</a>and-<a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a>+<a href="https://apify.com/docs/actor" target="_blank">Apify Actor</a>documentation.## Usage on the Apify CloudYou can also develop your web scraping projectin an online code editor directly on the-<a href="https://www.apify.com" target="_blank">Apify Cloud</a>.+<a href="https://apify.com" target="_blank">Apify Cloud</a>.You'll need to have an Apify Account.Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>,page in the app, click <i>Create new</i> and then go to the<i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.For more information, view the-<a href="https://www.apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.+<a href="https://apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.diff --git a/docs/guides/what_is_an_actor.md b/docs/guides/what_is_an_actor.mdindex ae3dfc5b2..1ef273eb5 100644--- a/docs/guides/what_is_an_actor.md+++ b/docs/guides/what_is_an_actor.md@@ -2,7 +2,7 @@id: whatisanactortitle: What is an Actor----When you deploy your script to the <a href="https://www.apify.com/" target="_blank">Apify Cloud</a>,+When you deploy your script to the <a href="https://apify.com/" target="_blank">Apify Cloud</a>,it becomes an actor.An actor is a serverless microservice that accepts an input and produces an output.It can run for a few seconds, hours or even infinitely.@@ -10,14 +10,14 @@ An actor can perform anything from a simple action such as filling out a web forto complex operations such as crawling an entire website and removing duplicates from a large dataset.To run an actor, you need to have an <a href="https://my.apify.com/)" target="_blank">Apify Account</a>.-Actors can be shared in the <a href="https://www.apify.com/library?&type=acts)" target="_blank">Apify Library</a>+Actors can be shared in the <a href="https://apify.com/library?&type=acts)" target="_blank">Apify Library</a>so that other people can use them.But don't worry, if you share your actor in the libraryand somebody uses it, it runs under their account, not yours.**Related links**-* <a href="https://www.apify.com/library?&type=acts" target="_blank">Library of existing actors</a>-* <a href="https://www.apify.com/docs/actor" target="_blank">Documentation</a>+* <a href="https://apify.com/library?&type=acts" target="_blank">Library of existing actors</a>+* <a href="https://apify.com/docs/actor" target="_blank">Documentation</a>* <a href="https://my.apify.com/actors" target="_blank">View actors in Apify app</a>-* <a href="https://www.apify.com/docs/api/v2#/reference/actors" target="_blank">API reference</a>+* <a href="https://apify.com/docs/api/v2#/reference/actors" target="_blank">API reference</a>diff --git a/examples/call_actor.js b/examples/call_actor.jsindex 5b0fd2bcc..a2d4ba7e1 100644--- a/examples/call_actor.js+++ b/examples/call_actor.js@@ -4,7 +4,7 @@* and how to call Apify API using* [`Apify.client`](../api/apify#module_Apify.client).* The script extracts the current Bitcoin prices from <a href="https://www.kraken.com/" target="_blank">Kraken.com</a>- * and sends them to your email using the <a href="https://www.apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.+ * and sends them to your email using the <a href="https://apify.com/apify/send-mail" target="_blank">apify/send-mail</a> actor.** To make the example work, you'll need an <a href="https://my.apify.com/" target="_blank">Apify Account</a>.* Go to <a href="https://my.apify.com/account#/integrations" target="_blank">Account - Integrations</a> page to obtain your API token@@ -35,7 +35,7 @@ Apify.main(async () => {// have available on the platform under the name: apify/send-mail.// The second parameter to the Apify.call() invocation is the actor's// desired input. You can find the required input parameters by checking- // the actor's documentation page: https://www.apify.com/apify/send-mail+ // the actor's documentation page: https://apify.com/apify/send-mailconsole.log(`Sending email to ${user.email}...`);await Apify.call('apify/send-mail', {to: user.email,diff --git a/examples/puppeteer_with_proxy.js b/examples/puppeteer_with_proxy.jsindex 490c4777f..5f894895f 100644--- a/examples/puppeteer_with_proxy.js+++ b/examples/puppeteer_with_proxy.js@@ -1,6 +1,6 @@/*** This example demonstrates how to load pages in headless Chrome / Puppeteer- * over <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ * over <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.* To make it work, you'll need an Apify Account* that has access to the proxy.* The proxy password is available on the <a href="https://my.apify.com/proxy" target="_blank">Proxy</a> page in the app.diff --git a/examples/screenshots.js b/examples/screenshots.jsindex ed0865409..4daa19ed8 100644--- a/examples/screenshots.js+++ b/examples/screenshots.js@@ -17,8 +17,8 @@** On the Apify cloud, the input can be either set manually* in the UI app or passed as the POST payload to the- * <a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>.- * For more details, see <a href="https://www.apify.com/docs/actor#input-output" target="_blank">Input and output</a>+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor API call</a>.+ * For more details, see <a href="https://apify.com/docs/actor#input-output" target="_blank">Input and output</a>* in the Apify Actor documentation.** To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base imagediff --git a/examples/synchronous_run.js b/examples/synchronous_run.jsindex d093a5aa4..299a75c52 100644--- a/examples/synchronous_run.js+++ b/examples/synchronous_run.js@@ -5,9 +5,9 @@** This actor* can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse, using the- * <a href="https://www.apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input" target="_blank">Run actor synchronously</a>* Apify API endpoint. The example is also shared as the- * <a href="https://www.apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>+ * <a href="https://apify.com/apify/example-golden-gate-webcam" target="_blank">apify/example-golden-gate-webcam</a>* actor in the Apify library, so you can test it directly there simply by sending a POST request to* ```http* https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]diff --git a/src/actor.js b/src/actor.jsindex bb9ecde53..3d554df64 100644--- a/src/actor.js+++ b/src/actor.js@@ -101,7 +101,7 @@ const waitForRunToFinish = async ({ actId, runId, token, waitSecs, taskId }) =>* }* ```* For the list of the `APIFY_XXX` environment variables, see- * <a href="https://www.apify.com/docs/actor#run-env-vars" target="_blank">Actor documentation</a>.+ * <a href="https://apify.com/docs/actor#run-env-vars" target="_blank">Actor documentation</a>.* If some of the variables are not defined or are invalid, the corresponding value in the resulting object will be null.** @returns {Object}@@ -262,7 +262,7 @@ let callMemoryWarningIssued = false;* [`Apify.callTask()`](../api/apify#module_Apify.callTask) function instead.** For more information about actors, read the- * <a href="https://www.apify.com/docs/actor" target="_blank">documentation</a>.+ * <a href="https://apify.com/docs/actor" target="_blank">documentation</a>.** **Example usage:***@@ -272,7 +272,7 @@ let callMemoryWarningIssued = false;* ```** Internally, the `call()` function invokes the- * <a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Run actor</a>* and several other API endpoints to obtain the output.** @param {String} actId@@ -407,7 +407,7 @@ export const call = async (actId, input, options = {}) => {* If you want to run an actor directly rather than an actor task, please use the* [`Apify.call()`](../api/apify#module_Apify.call) function instead.*- * For more information about actor tasks, read the [`documentation`](https://www.apify.com/docs/tasks).+ * For more information about actor tasks, read the [`documentation`](https://apify.com/docs/tasks).** **Example usage:***@@ -417,7 +417,7 @@ export const call = async (actId, input, options = {}) => {* ```** Internally, the `callTask()` function calls the- * <a href="https://www.apify.com/docs/api/v2#/reference/actor-tasks/runs-collection/run-task-asynchronously" target="_blank">Run task</a>+ * <a href="https://apify.com/docs/api/v2#/reference/actor-tasks/runs-collection/run-task-asynchronously" target="_blank">Run task</a>* and several other API endpoints to obtain the output.** @param {String} taskId@@ -490,10 +490,10 @@ export const callTask = async (taskId, input, options = {}) => {* [`Apify.call()`](../api/apify#module_Apify.call) or [`Apify.callTask()`](../api/apify#module_Apify.callTask) function.* The object is almost equivalent to the JSON response* of the- * <a href="https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>+ * <a href="https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor" target="_blank">Actor run</a>* Apify API endpoint and extended with certain fields.* For more details, see- * <a href="https://www.apify.com/docs/actor#run" target="_blank">Runs.</a>+ * <a href="https://apify.com/docs/actor#run" target="_blank">Runs.</a>** @typedef {Object} ActorRun* @property {String} id@@ -506,7 +506,7 @@ export const callTask = async (taskId, input, options = {}) => {* Time when the actor run finished. Contains `null` for running actors.* @property {String} status* Status of the run. For possible values, see- * <a href="https://www.apify.com/docs/actor#run-lifecycle" target="_blank">Run lifecycle</a>+ * <a href="https://apify.com/docs/actor#run-lifecycle" target="_blank">Run lifecycle</a>* in Apify actor documentation.* @property {Object} meta* Actor run meta-data. For example:@@ -539,7 +539,7 @@ export const callTask = async (taskId, input, options = {}) => {* ```* @property {String} buildId* ID of the actor build used for the run. For details, see- * <a href="https://www.apify.com/docs/actor#build" target="_blank">Builds</a>+ * <a href="https://apify.com/docs/actor#build" target="_blank">Builds</a>* in Apify actor documentation.* @property {String} buildNumber* Number of the actor build used for the run. For example, `0.0.10`.@@ -554,7 +554,7 @@ export const callTask = async (taskId, input, options = {}) => {* @property {String} containerUrl* URL on which the web server running inside actor run's Docker container can be accessed.* For more details, see- * <a href="https://www.apify.com/docs/actor#container-web-server" target="_blank">Container web server</a>+ * <a href="https://apify.com/docs/actor#container-web-server" target="_blank">Container web server</a>* in Apify actor documentation.* @property {Object} output* Contains output of the actor run. The value is `null` or `undefined` in case the actor is still running,@@ -579,7 +579,7 @@ export const callTask = async (taskId, input, options = {}) => {** For more information, see* the <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> page in the app- * or the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>.+ * or the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a>.** @param {Object} options* Object with the settings below:diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex 68e8ec3b5..3bd9a1fb9 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -150,10 +150,10 @@ const DEFAULT_OPTIONS = {* @param {Boolean} [options.useApifyProxy=false]* If set to `true`, `CheerioCrawler` will be configured to use* <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>+ * For more information, see the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a>* @param {String[]} [options.apifyProxyGroups]* An array of proxy groups to be used- * by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ * by the <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.* Only applied if the `useApifyProxy` option is `true`.* @param {String} [options.apifyProxySession]* Apify Proxy session identifier to be used with requests made by `CheerioCrawler`.diff --git a/src/dataset.js b/src/dataset.jsindex f9121d367..b707bf53b 100644--- a/src/dataset.js+++ b/src/dataset.js@@ -118,7 +118,7 @@ export const chunkBySize = (items, limitBytes) => {* Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the item in the dataset.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://www.apify.com/docs/storage#dataset" target="_blank">Apify Dataset</a>+ * <a href="https://apify.com/docs/storage#dataset" target="_blank">Apify Dataset</a>* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`* option to [`Apify.openDataset()`](apify#module_Apify.openDataset) function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.@@ -256,9 +256,9 @@ export class Dataset {* Returns an object containing general information about the dataset.** The function returns the same object as the Apify API Client's- * [getDataset](https://www.apify.com/docs/api/apify-client-js/latest#ApifyClient-datasets-getDataset)+ * [getDataset](https://apify.com/docs/api/apify-client-js/latest#ApifyClient-datasets-getDataset)* function, which in turn calls the- * [Get dataset](https://www.apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)+ * [Get dataset](https://apify.com/docs/api/v2#/reference/datasets/dataset/get-dataset)* API endpoint.** **Example:**diff --git a/src/key_value_store.js b/src/key_value_store.jsindex f392e6a42..c0ac30a71 100644--- a/src/key_value_store.js+++ b/src/key_value_store.js@@ -125,7 +125,7 @@ export const maybeStringify = (value, options) => {* The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://www.apify.com/docs/storage#key-value-store" target="_blank">Apify Key-value store</a>+ * <a href="https://apify.com/docs/storage#key-value-store" target="_blank">Apify Key-value store</a>* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`* option to [`Apify.openKeyValueStore()`](apify#module_Apify.openKeyValueStore) function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.diff --git a/src/puppeteer.js b/src/puppeteer.jsindex 34fcd6a59..6d93757b9 100644--- a/src/puppeteer.js+++ b/src/puppeteer.js@@ -51,10 +51,10 @@ const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {* @property {Boolean} [useApifyProxy=false]* If set to `true`, Puppeteer will be configured to use* <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.- * For more information, see the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a>+ * For more information, see the <a href="https://apify.com/docs/proxy" target="_blank">documentation</a>* @property {String[]} [apifyProxyGroups]* An array of proxy groups to be used- * by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ * by the <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>.* Only applied if the `useApifyProxy` option is `true`.* @property {String} [apifyProxySession]* Apify Proxy session identifier to be used by all the Chrome browsers.@@ -163,7 +163,7 @@ const getPuppeteerOrThrow = () => {* </li>* <li>* If <code>options.useApifyProxy</code> is <code>true</code> then the function generates a URL of- * <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>+ * <a href="https://apify.com/docs/proxy" target="_blank">Apify Proxy</a>* based on <code>options.apifyProxyGroups</code> and <code>options.apifyProxySession</code> and passes it as <code>options.proxyUrl</code>.* </li>* <li>@@ -184,7 +184,7 @@ const getPuppeteerOrThrow = () => {* NPM package installed in your project.* When running on the Apify cloud, you can achieve that simply* by using the `apify/actor-node-chrome` base Docker image for your actor - see- * <a href="https://www.apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>+ * <a href="https://apify.com/docs/actor#base-images" target="_blank">Apify Actor documentation</a>* for details.** For an example of usage, see the [Synchronous run Example](../examples/synchronousrun)diff --git a/src/request_queue.js b/src/request_queue.jsindex b49616f95..70ed9ff98 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -159,7 +159,7 @@ const getRequestId = (uniqueKey) => {* and `{NUMBER}` is an integer indicating the position of the request in the queue.** If the `APIFY_TOKEN` environment variable is set but `APIFY_LOCAL_STORAGE_DIR` not, the data is stored in the- * <a href="https://www.apify.com/docs/storage#queue" target="_blank">Apify Request Queue</a>+ * <a href="https://apify.com/docs/storage#queue" target="_blank">Apify Request Queue</a>* cloud storage. Note that you can force usage of the cloud storage also by passing the `forceCloud`* option to [`Apify.openRequestQueue()`](apify#module_Apify.openRequestQueue) function,* even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.@@ -559,9 +559,9 @@ export class RequestQueue {* Returns an object containing general information about the request queue.** The function returns the same object as the Apify API Client's- * [getQueue](https://www.apify.com/docs/api/apify-client-js/latest#ApifyClient-requestQueues-getQueue)+ * [getQueue](https://apify.com/docs/api/apify-client-js/latest#ApifyClient-requestQueues-getQueue)* function, which in turn calls the- * [Get request queue](https://www.apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)+ * [Get request queue](https://apify.com/docs/api/v2#/reference/request-queues/queue/get-request-queue)* API endpoint.** **Example:**diff --git a/src/utils.js b/src/utils.jsindex fdc3669c0..4b37d9605 100644--- a/src/utils.js+++ b/src/utils.js@@ -70,7 +70,7 @@ export const logSystemInfo = () => {/*** Gets the default instance of the `ApifyClient` class provided- * <a href="https://www.apify.com/docs/sdk/apify-client-js/latest"+ * <a href="https://apify.com/docs/sdk/apify-client-js/latest"* target="_blank">apify-client</a> by the NPM package.* The instance is created automatically by the Apify SDK and it is configured using the* `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN` environment variables.@@ -78,7 +78,7 @@ export const logSystemInfo = () => {* The instance is used for all underlying calls to the Apify API in functions such as* [`Apify.getValue()`](#module_Apify.getValue) or [`Apify.call()`](#module_Apify.call).* The settings of the client can be globally altered by calling the- * <a href="https://www.apify.com/docs/sdk/apify-client-js/latest#ApifyClient-setOptions"+ * <a href="https://apify.com/docs/sdk/apify-client-js/latest#ApifyClient-setOptions"* target="_blank">`Apify.client.setOptions()`</a> function.* Beware that altering these settings might have unintended effects on the entire Apify SDK package.*diff --git a/src/webdriver.js b/src/webdriver.jsindex 620a07c68..ab53cdeaf 100644--- a/src/webdriver.js+++ b/src/webdriver.js@@ -289,10 +289,10 @@ export const browse = (url, options) => {* To use this function, you need to have Google Chrome and* <a href="https://sites.google.com/a/chromium.org/chromedriver/" target="_blank">ChromeDriver</a> installed in your environment.* For example, you can use the `apify/actor-node-chrome` base Docker image for your actor - see- * <a href="https://www.apify.com/docs/actor#base-images" target="_blank">documentation</a>+ * <a href="https://apify.com/docs/actor#base-images" target="_blank">documentation</a>* for more details.*- * For an example of usage, see the <a href="https://www.apify.com/apify/example-selenium" target="_blank">apify/example-selenium</a> actor.+ * For an example of usage, see the <a href="https://apify.com/apify/example-selenium" target="_blank">apify/example-selenium</a> actor.** @param {Object} [options] Optional settings passed to WebDriver. Additionally the object can contain the following fields:* @param {String} [options.proxyUrl] - URL to a proxy server. Currently only `http://` scheme is supported.