Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: seo changes (#2588)
Saurav Jain3eebc73045f9
diff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl_all_links.mdxindex 85bf49471..371b834ca 100644--- a/docs/examples/crawl_all_links.mdx+++ b/docs/examples/crawl_all_links.mdx@@ -13,7 +13,7 @@ import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteerimport PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`-as the crawler navigates from page to page.+as the crawler navigates from page to page. This example can also be used to find all URLs on a domain by removing the `maxRequestsPerCrawl` option.:::tipdiff --git a/docs/examples/crawl_sitemap.mdx b/docs/examples/crawl_sitemap.mdxindex 711b3c337..fb80ac896 100644--- a/docs/examples/crawl_sitemap.mdx+++ b/docs/examples/crawl_sitemap.mdx@@ -12,7 +12,7 @@ import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';-This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.+We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.<Tabs groupId="crawler-type">diff --git a/docs/examples/crawler-plugins/index.mdx b/docs/examples/crawler-plugins/index.mdxindex fc38fa195..38699b23b 100644--- a/docs/examples/crawler-plugins/index.mdx+++ b/docs/examples/crawler-plugins/index.mdx@@ -1,6 +1,6 @@----id: playwright-puppeteer-extra-title: Using puppeteer-extra and playwright-extra+id: playwright-extra-puppeteer-stealth+title: Using Puppeteer Stealth Plugin (puppeteer-extra) and playwright-extra---import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';diff --git a/docs/examples/forms.mdx b/docs/examples/forms.mdxindex 9d3250afa..477180f9a 100644--- a/docs/examples/forms.mdx+++ b/docs/examples/forms.mdx@@ -1,6 +1,6 @@---id: forms-title: Forms+title: Fill and Submit a Form using Puppeteer---import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';diff --git a/docs/examples/http_crawler.mdx b/docs/examples/http_crawler.mdxindex 41f5d2539..39559ccc6 100644--- a/docs/examples/http_crawler.mdx+++ b/docs/examples/http_crawler.mdx@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';-This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.+This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.<RunnableCodeBlock className="language-js" type="cheerio">{HttpCrawlerSource}diff --git a/docs/examples/puppeteer_capture_screenshot.mdx b/docs/examples/puppeteer_capture_screenshot.mdxindex b9387e868..7cc302f50 100644--- a/docs/examples/puppeteer_capture_screenshot.mdx+++ b/docs/examples/puppeteer_capture_screenshot.mdx@@ -71,5 +71,7 @@ Using the context-aware <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#</TabItem></Tabs>+To take full page screenshot using puppeteer we need to pass parameter `fullPage` as `true`in the `screenshot()`: `page.screenshot(fullPage: true)`+In both examples using `page.screenshot()`, a `key` variable is created based on the URL of the web page. This variable is used as the key when savingeach screenshot into a key-value store.diff --git a/docs/guides/cheerio_crawler.mdx b/docs/guides/cheerio_crawler.mdxindex d6926e990..345d3a180 100644--- a/docs/guides/cheerio_crawler.mdx+++ b/docs/guides/cheerio_crawler.mdx@@ -62,7 +62,7 @@ This is not to show that Cheerio is better than plain browser JavaScript. Some m- May easily overload the target website with requests- Does not enable any manipulation of the website before scraping-## Example use of Cheerio+## Web scraping with Cheerio: Examples### Get text content of an elementdiff --git a/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx b/website/versioned_docs/version-3.11/examples/crawl_all_links.mdxindex 85bf49471..371b834ca 100644--- a/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx+++ b/website/versioned_docs/version-3.11/examples/crawl_all_links.mdx@@ -13,7 +13,7 @@ import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteerimport PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`-as the crawler navigates from page to page.+as the crawler navigates from page to page. This example can also be used to find all URLs on a domain by removing the `maxRequestsPerCrawl` option.:::tipdiff --git a/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx b/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdxindex 711b3c337..fb80ac896 100644--- a/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx+++ b/website/versioned_docs/version-3.11/examples/crawl_sitemap.mdx@@ -12,7 +12,7 @@ import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';-This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.+We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.<Tabs groupId="crawler-type">diff --git a/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx b/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdxindex fc38fa195..38699b23b 100644--- a/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx+++ b/website/versioned_docs/version-3.11/examples/crawler-plugins/index.mdx@@ -1,6 +1,6 @@----id: playwright-puppeteer-extra-title: Using puppeteer-extra and playwright-extra+id: playwright-extra-puppeteer-stealth+title: Using Puppeteer Stealth Plugin (puppeteer-extra) and playwright-extra---import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';diff --git a/website/versioned_docs/version-3.11/examples/forms.mdx b/website/versioned_docs/version-3.11/examples/forms.mdxindex 9d3250afa..eac463261 100644--- a/website/versioned_docs/version-3.11/examples/forms.mdx+++ b/website/versioned_docs/version-3.11/examples/forms.mdx@@ -1,6 +1,6 @@---id: forms-title: Forms+title: Fill and Submit a Form using Puppeteers---import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';diff --git a/website/versioned_docs/version-3.11/examples/http_crawler.mdx b/website/versioned_docs/version-3.11/examples/http_crawler.mdxindex 41f5d2539..39559ccc6 100644--- a/website/versioned_docs/version-3.11/examples/http_crawler.mdx+++ b/website/versioned_docs/version-3.11/examples/http_crawler.mdx@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';-This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.+This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.<RunnableCodeBlock className="language-js" type="cheerio">{HttpCrawlerSource}diff --git a/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx b/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdxindex b9387e868..7cc302f50 100644--- a/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx+++ b/website/versioned_docs/version-3.11/examples/puppeteer_capture_screenshot.mdx@@ -71,5 +71,7 @@ Using the context-aware <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#</TabItem></Tabs>+To take full page screenshot using puppeteer we need to pass parameter `fullPage` as `true`in the `screenshot()`: `page.screenshot(fullPage: true)`+In both examples using `page.screenshot()`, a `key` variable is created based on the URL of the web page. This variable is used as the key when savingeach screenshot into a key-value store.diff --git a/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx b/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdxindex d6926e990..345d3a180 100644--- a/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx+++ b/website/versioned_docs/version-3.11/guides/cheerio_crawler.mdx@@ -62,7 +62,7 @@ This is not to show that Cheerio is better than plain browser JavaScript. Some m- May easily overload the target website with requests- Does not enable any manipulation of the website before scraping-## Example use of Cheerio+## Web scraping with Cheerio: Examples### Get text content of an element