Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: seo changes (#2588)

Saurav Jain3eebc73045f9
diff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl_all_links.mdx
index 85bf49471..371b834ca 100644
--- a/docs/examples/crawl_all_links.mdx
+++ b/docs/examples/crawl_all_links.mdx
@@ -13,7 +13,7 @@ import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteer
import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';
This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`
-as the crawler navigates from page to page.
+as the crawler navigates from page to page. This example can also be used to find all URLs on a domain by removing the `maxRequestsPerCrawl` option.
:::tip
diff --git a/docs/examples/crawl_sitemap.mdx b/docs/examples/crawl_sitemap.mdx
index 711b3c337..fb80ac896 100644
--- a/docs/examples/crawl_sitemap.mdx
+++ b/docs/examples/crawl_sitemap.mdx
@@ -12,7 +12,7 @@ import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';
import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';
import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';
-This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.
+We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.
<Tabs groupId="crawler-type">
diff --git a/docs/examples/crawler-plugins/index.mdx b/docs/examples/crawler-plugins/index.mdx
index fc38fa195..38699b23b 100644
--- a/docs/examples/crawler-plugins/index.mdx
+++ b/docs/examples/crawler-plugins/index.mdx
@@ -1,6 +1,6 @@
---
-id: playwright-puppeteer-extra
-title: Using puppeteer-extra and playwright-extra
+id: playwright-extra-puppeteer-stealth
+title: Using Puppeteer Stealth Plugin (puppeteer-extra) and playwright-extra
---
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
diff --git a/docs/examples/forms.mdx b/docs/examples/forms.mdx
index 9d3250afa..477180f9a 100644
--- a/docs/examples/forms.mdx
+++ b/docs/examples/forms.mdx
@@ -1,6 +1,6 @@
---
id: forms
-title: Forms
+title: Fill and Submit a Form using Puppeteer
---
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
diff --git a/docs/examples/http_crawler.mdx b/docs/examples/http_crawler.mdx
index 41f5d2539..39559ccc6 100644
--- a/docs/examples/http_crawler.mdx
+++ b/docs/examples/http_crawler.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';
-This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.
+This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.
<RunnableCodeBlock className="language-js" type="cheerio">
{HttpCrawlerSource}
diff --git a/docs/examples/puppeteer_capture_screenshot.mdx b/docs/examples/puppeteer_capture_screenshot.mdx
index b9387e868..7cc302f50 100644
--- a/docs/examples/puppeteer_capture_screenshot.mdx
+++ b/docs/examples/puppeteer_capture_screenshot.mdx
@@ -71,5 +71,7 @@ Using the context-aware <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#
</TabItem>
</Tabs>
+To take full page screenshot using puppeteer we need to pass parameter `fullPage` as `true`in the `screenshot()`: `page.screenshot(fullPage: true)`
+
In both examples using `page.screenshot()`, a `key` variable is created based on the URL of the web page. This variable is used as the key when saving
each screenshot into a key-value store.
diff --git a/docs/guides/cheerio_crawler.mdx b/docs/guides/cheerio_crawler.mdx
index d6926e990..345d3a180 100644
--- a/docs/guides/cheerio_crawler.mdx
+++ b/docs/guides/cheerio_crawler.mdx
@@ -62,7 +62,7 @@ This is not to show that Cheerio is better than plain browser JavaScript. Some m
- May easily overload the target website with requests
- Does not enable any manipulation of the website before scraping
-## Example use of Cheerio
+## Web scraping with Cheerio: Examples
### Get text content of an element
diff --git a/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx b/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx
index 85bf49471..371b834ca 100644
--- a/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx
+++ b/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx
@@ -13,7 +13,7 @@ import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteer
import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';
This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`
-as the crawler navigates from page to page.
+as the crawler navigates from page to page. This example can also be used to find all URLs on a domain by removing the `maxRequestsPerCrawl` option.
:::tip
diff --git a/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx b/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx
index 711b3c337..fb80ac896 100644
--- a/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx
+++ b/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx
@@ -12,7 +12,7 @@ import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';
import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';
import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';
-This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.
+We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.
<Tabs groupId="crawler-type">
diff --git a/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx b/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx
index fc38fa195..38699b23b 100644
--- a/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx
+++ b/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx
@@ -1,6 +1,6 @@
---
-id: playwright-puppeteer-extra
-title: Using puppeteer-extra and playwright-extra
+id: playwright-extra-puppeteer-stealth
+title: Using Puppeteer Stealth Plugin (puppeteer-extra) and playwright-extra
---
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
diff --git a/website/versioned_docs/version-3.11/examples/forms.mdx b/website/versioned_docs/version-3.11/examples/forms.mdx
index 9d3250afa..eac463261 100644
--- a/website/versioned_docs/version-3.11/examples/forms.mdx
+++ b/website/versioned_docs/version-3.11/examples/forms.mdx
@@ -1,6 +1,6 @@
---
id: forms
-title: Forms
+title: Fill and Submit a Form using Puppeteers
---
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
diff --git a/website/versioned_docs/version-3.11/examples/http_crawler.mdx b/website/versioned_docs/version-3.11/examples/http_crawler.mdx
index 41f5d2539..39559ccc6 100644
--- a/website/versioned_docs/version-3.11/examples/http_crawler.mdx
+++ b/website/versioned_docs/version-3.11/examples/http_crawler.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';
-This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.
+This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.
<RunnableCodeBlock className="language-js" type="cheerio">
{HttpCrawlerSource}
diff --git a/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx b/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx
index b9387e868..7cc302f50 100644
--- a/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx
+++ b/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx
@@ -71,5 +71,7 @@ Using the context-aware <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#
</TabItem>
</Tabs>
+To take full page screenshot using puppeteer we need to pass parameter `fullPage` as `true`in the `screenshot()`: `page.screenshot(fullPage: true)`
+
In both examples using `page.screenshot()`, a `key` variable is created based on the URL of the web page. This variable is used as the key when saving
each screenshot into a key-value store.
diff --git a/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx b/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx
index d6926e990..345d3a180 100644
--- a/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx
+++ b/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx
@@ -62,7 +62,7 @@ This is not to show that Cheerio is better than plain browser JavaScript. Some m
- May easily overload the target website with requests
- Does not enable any manipulation of the website before scraping
-## Example use of Cheerio
+## Web scraping with Cheerio: Examples
### Get text content of an element