Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: rename docs files to kebab-case (#4235)
Martin Adámekade1625438eb
diff --git a/docs/deployment/apify_platform_init_exit.ts b/docs/deployment/apify-platform-init-exit.tssimilarity index 100%rename from docs/deployment/apify_platform_init_exit.tsrename to docs/deployment/apify-platform-init-exit.tsdiff --git a/docs/deployment/apify_platform_main.ts b/docs/deployment/apify-platform-main.tssimilarity index 100%rename from docs/deployment/apify_platform_main.tsrename to docs/deployment/apify-platform-main.tsdiff --git a/docs/deployment/apify_platform.mdx b/docs/deployment/apify-platform.mdxsimilarity index 99%rename from docs/deployment/apify_platform.mdxrename to docs/deployment/apify-platform.mdxindex a5998bc93..0204a268f 100644--- a/docs/deployment/apify_platform.mdx+++ b/docs/deployment/apify-platform.mdx@@ -10,8 +10,8 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import MainSource from '!!raw-loader!./apify_platform_main.ts';-import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';+import MainSource from '!!raw-loader!./apify-platform-main.ts';+import InitExitSource from '!!raw-loader!./apify-platform-init-exit.ts';Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scrapingand automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),diff --git a/docs/examples/add_data_to_dataset.mdx b/docs/examples/add-data-to-dataset.mdxsimilarity index 96%rename from docs/examples/add_data_to_dataset.mdxrename to docs/examples/add-data-to-dataset.mdxindex 0bf29a3d8..c67c5f91b 100644--- a/docs/examples/add_data_to_dataset.mdx+++ b/docs/examples/add-data-to-dataset.mdx@@ -5,7 +5,7 @@ title: Add data to datasetimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import AddDataToDatasetSource from '!!raw-loader!roa-loader!./add_data_to_dataset.ts';+import AddDataToDatasetSource from '!!raw-loader!roa-loader!./add-data-to-dataset.ts';This example saves data to the default dataset. If the dataset doesn't exist, it will be created.You can save data to custom datasets by using <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink>diff --git a/docs/examples/add_data_to_dataset.ts b/docs/examples/add-data-to-dataset.tssimilarity index 100%rename from docs/examples/add_data_to_dataset.tsrename to docs/examples/add-data-to-dataset.tsdiff --git a/docs/examples/basic_crawler.mdx b/docs/examples/basic-crawler.mdxsimilarity index 94%rename from docs/examples/basic_crawler.mdxrename to docs/examples/basic-crawler.mdxindex 81a53ee9f..caeadd590 100644--- a/docs/examples/basic_crawler.mdx+++ b/docs/examples/basic-crawler.mdx@@ -5,7 +5,7 @@ title: Basic crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import BasicCrawlerSource from '!!raw-loader!roa-loader!./basic_crawler.ts';+import BasicCrawlerSource from '!!raw-loader!roa-loader!./basic-crawler.ts';This is the most bare-bones example of using Crawlee, which demonstrates some of its building blocks such as the <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>. You probably don't need to go this deep though, and it would be better to start with one of the full-featured crawlerslike <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> or <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.diff --git a/docs/examples/basic_crawler.ts b/docs/examples/basic-crawler.tssimilarity index 100%rename from docs/examples/basic_crawler.tsrename to docs/examples/basic-crawler.tsdiff --git a/docs/examples/cheerio_crawler.mdx b/docs/examples/cheerio-crawler.mdxsimilarity index 97%rename from docs/examples/cheerio_crawler.mdxrename to docs/examples/cheerio-crawler.mdxindex 434488ae2..7b73eea15 100644--- a/docs/examples/cheerio_crawler.mdx+++ b/docs/examples/cheerio-crawler.mdx@@ -5,7 +5,7 @@ title: Cheerio crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CheerioCrawlerSource from '!!raw-loader!roa-loader!./cheerio_crawler.ts';+import CheerioCrawlerSource from '!!raw-loader!roa-loader!./cheerio-crawler.ts';This example demonstrates how to use <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> to crawl a list of URLs from an external file, load each URL using a plain HTTP request, parse the HTML using the [Cheerio library](https://www.npmjs.com/package/cheerio) and extract some data from it: the page title and all `h1` tags.diff --git a/docs/examples/cheerio_crawler.ts b/docs/examples/cheerio-crawler.tssimilarity index 100%rename from docs/examples/cheerio_crawler.tsrename to docs/examples/cheerio-crawler.tsdiff --git a/docs/examples/crawl_all_links_cheerio.ts b/docs/examples/crawl-all-links-cheerio.tssimilarity index 100%rename from docs/examples/crawl_all_links_cheerio.tsrename to docs/examples/crawl-all-links-cheerio.tsdiff --git a/docs/examples/crawl_all_links_playwright.ts b/docs/examples/crawl-all-links-playwright.tssimilarity index 100%rename from docs/examples/crawl_all_links_playwright.tsrename to docs/examples/crawl-all-links-playwright.tsdiff --git a/docs/examples/crawl_all_links_puppeteer.ts b/docs/examples/crawl-all-links-puppeteer.tssimilarity index 100%rename from docs/examples/crawl_all_links_puppeteer.tsrename to docs/examples/crawl-all-links-puppeteer.tsdiff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl-all-links.mdxsimilarity index 86%rename from docs/examples/crawl_all_links.mdxrename to docs/examples/crawl-all-links.mdxindex babe7faa3..72087eb54 100644--- a/docs/examples/crawl_all_links.mdx+++ b/docs/examples/crawl-all-links.mdx@@ -8,9 +8,9 @@ import TabItem from '@theme/TabItem';import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CheerioSource from '!!raw-loader!roa-loader!./crawl_all_links_cheerio.ts';-import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_all_links_puppeteer.ts';-import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_all_links_playwright.ts';+import CheerioSource from '!!raw-loader!roa-loader!./crawl-all-links-cheerio.ts';+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl-all-links-puppeteer.ts';+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl-all-links-playwright.ts';This example uses the `enqueueLinks()` method to add new links to the `RequestQueue`as the crawler navigates from page to page. This example can also be used to find all URLs on a domain by removing the `maxRequestsPerCrawl` option.diff --git a/docs/examples/crawl_multiple_urls_cheerio.ts b/docs/examples/crawl-multiple-urls-cheerio.tssimilarity index 100%rename from docs/examples/crawl_multiple_urls_cheerio.tsrename to docs/examples/crawl-multiple-urls-cheerio.tsdiff --git a/docs/examples/crawl_multiple_urls_playwright.ts b/docs/examples/crawl-multiple-urls-playwright.tssimilarity index 100%rename from docs/examples/crawl_multiple_urls_playwright.tsrename to docs/examples/crawl-multiple-urls-playwright.tsdiff --git a/docs/examples/crawl_multiple_urls_puppeteer.ts b/docs/examples/crawl-multiple-urls-puppeteer.tssimilarity index 100%rename from docs/examples/crawl_multiple_urls_puppeteer.tsrename to docs/examples/crawl-multiple-urls-puppeteer.tsdiff --git a/docs/examples/crawl_multiple_urls.mdx b/docs/examples/crawl-multiple-urls.mdxsimilarity index 80%rename from docs/examples/crawl_multiple_urls.mdxrename to docs/examples/crawl-multiple-urls.mdxindex e6510cff9..3bb63c824 100644--- a/docs/examples/crawl_multiple_urls.mdx+++ b/docs/examples/crawl-multiple-urls.mdx@@ -7,9 +7,9 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';-import CheerioSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_cheerio.ts';-import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_puppeteer.ts';-import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_multiple_urls_playwright.ts';+import CheerioSource from '!!raw-loader!roa-loader!./crawl-multiple-urls-cheerio.ts';+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl-multiple-urls-puppeteer.ts';+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl-multiple-urls-playwright.ts';This example crawls the specified list of URLs.diff --git a/docs/examples/crawl_relative_links_all.ts b/docs/examples/crawl-relative-links-all.tssimilarity index 100%rename from docs/examples/crawl_relative_links_all.tsrename to docs/examples/crawl-relative-links-all.tsdiff --git a/docs/examples/crawl_relative_links_same_domain.ts b/docs/examples/crawl-relative-links-same-domain.tssimilarity index 100%rename from docs/examples/crawl_relative_links_same_domain.tsrename to docs/examples/crawl-relative-links-same-domain.tsdiff --git a/docs/examples/crawl_relative_links_same_hostname.ts b/docs/examples/crawl-relative-links-same-hostname.tssimilarity index 100%rename from docs/examples/crawl_relative_links_same_hostname.tsrename to docs/examples/crawl-relative-links-same-hostname.tsdiff --git a/docs/examples/crawl_relative_links.mdx b/docs/examples/crawl-relative-links.mdxsimilarity index 92%rename from docs/examples/crawl_relative_links.mdxrename to docs/examples/crawl-relative-links.mdxindex 7281f0c7a..0baee830f 100644--- a/docs/examples/crawl_relative_links.mdx+++ b/docs/examples/crawl-relative-links.mdx@@ -8,9 +8,9 @@ import TabItem from '@theme/TabItem';import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import AllLinksSource from '!!raw-loader!roa-loader!./crawl_relative_links_all.ts';-import SameDomainSource from '!!raw-loader!roa-loader!./crawl_relative_links_same_domain.ts';-import SameHostnameSource from '!!raw-loader!roa-loader!./crawl_relative_links_same_hostname.ts';+import AllLinksSource from '!!raw-loader!roa-loader!./crawl-relative-links-all.ts';+import SameDomainSource from '!!raw-loader!roa-loader!./crawl-relative-links-same-domain.ts';+import SameHostnameSource from '!!raw-loader!roa-loader!./crawl-relative-links-same-hostname.ts';When crawling a website, you may encounter different types of links present that you may want to crawl.To facilitate the easy crawling of such links, we provide the `enqueueLinks()` method on the crawler context, which willdiff --git a/docs/examples/crawl_single_url.mdx b/docs/examples/crawl-single-url.mdxsimilarity index 91%rename from docs/examples/crawl_single_url.mdxrename to docs/examples/crawl-single-url.mdxindex 99415225c..53441334b 100644--- a/docs/examples/crawl_single_url.mdx+++ b/docs/examples/crawl-single-url.mdx@@ -5,7 +5,7 @@ title: Crawl a single URLimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./crawl_single_url.ts';+import CrawlSource from '!!raw-loader!roa-loader!./crawl-single-url.ts';This example uses the [`got-scraping`](https://github.com/apify/got-scraping) npm packageto grab the HTML of a web page.diff --git a/docs/examples/crawl_single_url.ts b/docs/examples/crawl-single-url.tssimilarity index 100%rename from docs/examples/crawl_single_url.tsrename to docs/examples/crawl-single-url.tsdiff --git a/docs/examples/crawl_sitemap_cheerio.ts b/docs/examples/crawl-sitemap-cheerio.tssimilarity index 100%rename from docs/examples/crawl_sitemap_cheerio.tsrename to docs/examples/crawl-sitemap-cheerio.tsdiff --git a/docs/examples/crawl_sitemap_playwright.ts b/docs/examples/crawl-sitemap-playwright.tssimilarity index 100%rename from docs/examples/crawl_sitemap_playwright.tsrename to docs/examples/crawl-sitemap-playwright.tsdiff --git a/docs/examples/crawl_sitemap_puppeteer.ts b/docs/examples/crawl-sitemap-puppeteer.tssimilarity index 100%rename from docs/examples/crawl_sitemap_puppeteer.tsrename to docs/examples/crawl-sitemap-puppeteer.tsdiff --git a/docs/examples/crawl_sitemap.mdx b/docs/examples/crawl-sitemap.mdxsimilarity index 85%rename from docs/examples/crawl_sitemap.mdxrename to docs/examples/crawl-sitemap.mdxindex fb80ac896..e34c2dea2 100644--- a/docs/examples/crawl_sitemap.mdx+++ b/docs/examples/crawl-sitemap.mdx@@ -8,9 +8,9 @@ import TabItem from '@theme/TabItem';import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CheerioSource from '!!raw-loader!roa-loader!./crawl_sitemap_cheerio.ts';-import PuppeteerSource from '!!raw-loader!roa-loader!./crawl_sitemap_puppeteer.ts';-import PlaywrightSource from '!!raw-loader!roa-loader!./crawl_sitemap_playwright.ts';+import CheerioSource from '!!raw-loader!roa-loader!./crawl-sitemap-cheerio.ts';+import PuppeteerSource from '!!raw-loader!roa-loader!./crawl-sitemap-puppeteer.ts';+import PlaywrightSource from '!!raw-loader!roa-loader!./crawl-sitemap-playwright.ts';We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.diff --git a/docs/examples/crawl_some_links.mdx b/docs/examples/crawl-some-links.mdxsimilarity index 92%rename from docs/examples/crawl_some_links.mdxrename to docs/examples/crawl-some-links.mdxindex c239b237f..2047b971c 100644--- a/docs/examples/crawl_some_links.mdx+++ b/docs/examples/crawl-some-links.mdx@@ -5,7 +5,7 @@ title: Crawl some links on a websiteimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';+import CrawlSource from '!!raw-loader!roa-loader!./crawl-some-links.ts';This <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> example uses the <ApiLink to="basic-crawler/interface/EnqueueUrlsOptions#include">`include`</ApiLink> property in the <ApiLink to="cheerio-crawler/interface/CheerioCrawlingContext#enqueueLinks">`enqueueLinks()`</ApiLink> method to only add links to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> queue if they match the specified pattern.diff --git a/docs/examples/crawl_some_links.ts b/docs/examples/crawl-some-links.tssimilarity index 100%rename from docs/examples/crawl_some_links.tsrename to docs/examples/crawl-some-links.tsdiff --git a/docs/examples/export_entire_dataset.mdx b/docs/examples/export-entire-dataset.mdxsimilarity index 85%rename from docs/examples/export_entire_dataset.mdxrename to docs/examples/export-entire-dataset.mdxindex bea044b92..7205379f2 100644--- a/docs/examples/export_entire_dataset.mdx+++ b/docs/examples/export-entire-dataset.mdx@@ -5,7 +5,7 @@ title: Export entire dataset to one fileimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./export_entire_dataset.ts';+import CrawlSource from '!!raw-loader!roa-loader!./export-entire-dataset.ts';This `Dataset` example uses the `exportToValue` function to export the entire default dataset to a single CSV file into the default key-value store.diff --git a/docs/examples/export_entire_dataset.ts b/docs/examples/export-entire-dataset.tssimilarity index 100%rename from docs/examples/export_entire_dataset.tsrename to docs/examples/export-entire-dataset.tsdiff --git a/docs/examples/file_download_stream.mdx b/docs/examples/file-download-stream.mdxsimilarity index 90%rename from docs/examples/file_download_stream.mdxrename to docs/examples/file-download-stream.mdxindex bd7c130eb..e6bc739d9 100644--- a/docs/examples/file_download_stream.mdx+++ b/docs/examples/file-download-stream.mdx@@ -5,7 +5,7 @@ title: Download a file with Node.js streamsimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import FileDownloadSource from '!!raw-loader!roa-loader!./file_download_stream.ts';+import FileDownloadSource from '!!raw-loader!roa-loader!./file-download-stream.ts';For larger files, it is more efficient to use Node.js streams to download and transfer the files. This example demonstrates how to download files using streams.diff --git a/docs/examples/file_download_stream.ts b/docs/examples/file-download-stream.tssimilarity index 100%rename from docs/examples/file_download_stream.tsrename to docs/examples/file-download-stream.tsdiff --git a/docs/examples/file_download.mdx b/docs/examples/file-download.mdxsimilarity index 92%rename from docs/examples/file_download.mdxrename to docs/examples/file-download.mdxindex 090320eda..dd2061b48 100644--- a/docs/examples/file_download.mdx+++ b/docs/examples/file-download.mdx@@ -5,7 +5,7 @@ title: Download a fileimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import FileDownloadSource from '!!raw-loader!roa-loader!./file_download.ts';+import FileDownloadSource from '!!raw-loader!roa-loader!./file-download.ts';When webcrawling, you sometimes need to download files such as images, PDFs, or other binary files. This example demonstrates how to download files using Crawlee and save them to the default key-value store.diff --git a/docs/examples/file_download.ts b/docs/examples/file-download.tssimilarity index 100%rename from docs/examples/file_download.tsrename to docs/examples/file-download.tsdiff --git a/docs/examples/http_crawler.mdx b/docs/examples/http-crawler.mdxsimilarity index 89%rename from docs/examples/http_crawler.mdxrename to docs/examples/http-crawler.mdxindex 39559ccc6..eed02afb4 100644--- a/docs/examples/http_crawler.mdx+++ b/docs/examples/http-crawler.mdx@@ -5,7 +5,7 @@ title: HTTP crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import HttpCrawlerSource from '!!raw-loader!roa-loader!./http_crawler.ts';+import HttpCrawlerSource from '!!raw-loader!roa-loader!./http-crawler.ts';This example demonstrates how to use <ApiLink to="http-crawler/class/HttpCrawler">`HttpCrawler`</ApiLink> to build a HTML crawler that crawls a list of URLs from an external file, load each URL using a plain HTTP request, and save HTML.diff --git a/docs/examples/http_crawler.ts b/docs/examples/http-crawler.tssimilarity index 100%rename from docs/examples/http_crawler.tsrename to docs/examples/http-crawler.tsdiff --git a/docs/examples/map_and_reduce.mdx b/docs/examples/map-and-reduce.mdxsimilarity index 100%rename from docs/examples/map_and_reduce.mdxrename to docs/examples/map-and-reduce.mdxdiff --git a/docs/examples/playwright_crawler_firefox.mdx b/docs/examples/playwright-crawler-firefox.mdxsimilarity index 97%rename from docs/examples/playwright_crawler_firefox.mdxrename to docs/examples/playwright-crawler-firefox.mdxindex 08aa865f1..bb75f6d67 100644--- a/docs/examples/playwright_crawler_firefox.mdx+++ b/docs/examples/playwright-crawler-firefox.mdx@@ -5,7 +5,7 @@ title: Using Firefox browser with Playwright crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import PlaywrightFirefoxSource from '!!raw-loader!roa-loader!./playwright_crawler_firefox.ts';+import PlaywrightFirefoxSource from '!!raw-loader!roa-loader!./playwright-crawler-firefox.ts';This example demonstrates how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> with headless Firefox browser.diff --git a/docs/examples/playwright_crawler_firefox.ts b/docs/examples/playwright-crawler-firefox.tssimilarity index 100%rename from docs/examples/playwright_crawler_firefox.tsrename to docs/examples/playwright-crawler-firefox.tsdiff --git a/docs/examples/playwright_crawler.mdx b/docs/examples/playwright-crawler.mdxsimilarity index 94%rename from docs/examples/playwright_crawler.mdxrename to docs/examples/playwright-crawler.mdxindex c029c2b32..ce4418e53 100644--- a/docs/examples/playwright_crawler.mdx+++ b/docs/examples/playwright-crawler.mdx@@ -5,7 +5,7 @@ title: Playwright crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./playwright_crawler.ts';+import CrawlSource from '!!raw-loader!roa-loader!./playwright-crawler.ts';This example demonstrates how to use <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> in combination with <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> to recursively scrape the [Hacker News website](https://news.ycombinator.com) using headless Chrome / Playwright.diff --git a/docs/examples/playwright_crawler.ts b/docs/examples/playwright-crawler.tssimilarity index 100%rename from docs/examples/playwright_crawler.tsrename to docs/examples/playwright-crawler.tsdiff --git a/docs/examples/puppeteer_capture_screenshot.mdx b/docs/examples/puppeteer-capture-screenshot.mdxsimilarity index 92%rename from docs/examples/puppeteer_capture_screenshot.mdxrename to docs/examples/puppeteer-capture-screenshot.mdxindex 57de3aa9b..5b92c386d 100644--- a/docs/examples/puppeteer_capture_screenshot.mdx+++ b/docs/examples/puppeteer-capture-screenshot.mdx@@ -8,10 +8,10 @@ import TabItem from '@theme/TabItem';import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import PuppeteerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer_page_screenshot.ts';-import PuppeteerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_utils_snapshot.ts';-import PuppeteerCrawlerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_page_screenshot.ts';-import PuppeteerCrawlerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer_crawler_crawler_utils_snapshot.ts';+import PuppeteerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer-page-screenshot.ts';+import PuppeteerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer-crawler-utils-snapshot.ts';+import PuppeteerCrawlerPageScreenshotSource from '!!raw-loader!roa-loader!./puppeteer-crawler-page-screenshot.ts';+import PuppeteerCrawlerCrawlerUtilsSnapshotSource from '!!raw-loader!roa-loader!./puppeteer-crawler-crawler-utils-snapshot.ts';## Using Puppeteer directlydiff --git a/docs/examples/puppeteer_crawler_crawler_utils_snapshot.ts b/docs/examples/puppeteer-crawler-crawler-utils-snapshot.tssimilarity index 100%rename from docs/examples/puppeteer_crawler_crawler_utils_snapshot.tsrename to docs/examples/puppeteer-crawler-crawler-utils-snapshot.tsdiff --git a/docs/examples/puppeteer_crawler_page_screenshot.ts b/docs/examples/puppeteer-crawler-page-screenshot.tssimilarity index 100%rename from docs/examples/puppeteer_crawler_page_screenshot.tsrename to docs/examples/puppeteer-crawler-page-screenshot.tsdiff --git a/docs/examples/puppeteer_crawler_utils_snapshot.ts b/docs/examples/puppeteer-crawler-utils-snapshot.tssimilarity index 100%rename from docs/examples/puppeteer_crawler_utils_snapshot.tsrename to docs/examples/puppeteer-crawler-utils-snapshot.tsdiff --git a/docs/examples/puppeteer_crawler.mdx b/docs/examples/puppeteer-crawler.mdxsimilarity index 94%rename from docs/examples/puppeteer_crawler.mdxrename to docs/examples/puppeteer-crawler.mdxindex a2df07301..f78259b7c 100644--- a/docs/examples/puppeteer_crawler.mdx+++ b/docs/examples/puppeteer-crawler.mdx@@ -5,7 +5,7 @@ title: Puppeteer crawlerimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./puppeteer_crawler.ts';+import CrawlSource from '!!raw-loader!roa-loader!./puppeteer-crawler.ts';This example demonstrates how to use <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> in combinationwith <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>diff --git a/docs/examples/puppeteer_crawler.ts b/docs/examples/puppeteer-crawler.tssimilarity index 100%rename from docs/examples/puppeteer_crawler.tsrename to docs/examples/puppeteer-crawler.tsdiff --git a/docs/examples/puppeteer_page_screenshot.ts b/docs/examples/puppeteer-page-screenshot.tssimilarity index 100%rename from docs/examples/puppeteer_page_screenshot.tsrename to docs/examples/puppeteer-page-screenshot.tsdiff --git a/docs/examples/puppeteer_recursive_crawl.mdx b/docs/examples/puppeteer-recursive-crawl.mdxsimilarity index 87%rename from docs/examples/puppeteer_recursive_crawl.mdxrename to docs/examples/puppeteer-recursive-crawl.mdxindex cd5b21c0b..aa449e9f7 100644--- a/docs/examples/puppeteer_recursive_crawl.mdx+++ b/docs/examples/puppeteer-recursive-crawl.mdx@@ -5,7 +5,7 @@ title: Puppeteer recursive crawlimport RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';import ApiLink from '@site/src/components/ApiLink';-import CrawlSource from '!!raw-loader!roa-loader!./puppeteer_recursive_crawl.ts';+import CrawlSource from '!!raw-loader!roa-loader!./puppeteer-recursive-crawl.ts';Run the following example to perform a recursive crawl of a website using <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink>.diff --git a/docs/examples/puppeteer_recursive_crawl.ts b/docs/examples/puppeteer-recursive-crawl.tssimilarity index 100%rename from docs/examples/puppeteer_recursive_crawl.tsrename to docs/examples/puppeteer-recursive-crawl.tsdiff --git a/docs/guides/avoid_blocking_camoufox.ts b/docs/guides/avoid-blocking-camoufox.tssimilarity index 100%rename from docs/guides/avoid_blocking_camoufox.tsrename to docs/guides/avoid-blocking-camoufox.tsdiff --git a/docs/guides/avoid_blocking_playwright_fingerprints_off.ts b/docs/guides/avoid-blocking-playwright-fingerprints-off.tssimilarity index 100%rename from docs/guides/avoid_blocking_playwright_fingerprints_off.tsrename to docs/guides/avoid-blocking-playwright-fingerprints-off.tsdiff --git a/docs/guides/avoid_blocking_playwright.ts b/docs/guides/avoid-blocking-playwright.tssimilarity index 100%rename from docs/guides/avoid_blocking_playwright.tsrename to docs/guides/avoid-blocking-playwright.tsdiff --git a/docs/guides/avoid_blocking_puppeteer_fingerprints_off.ts b/docs/guides/avoid-blocking-puppeteer-fingerprints-off.tssimilarity index 100%rename from docs/guides/avoid_blocking_puppeteer_fingerprints_off.tsrename to docs/guides/avoid-blocking-puppeteer-fingerprints-off.tsdiff --git a/docs/guides/avoid_blocking_puppeteer.ts b/docs/guides/avoid-blocking-puppeteer.tssimilarity index 100%rename from docs/guides/avoid_blocking_puppeteer.tsrename to docs/guides/avoid-blocking-puppeteer.tsdiff --git a/docs/guides/avoid_blocking.mdx b/docs/guides/avoid-blocking.mdxsimilarity index 95%rename from docs/guides/avoid_blocking.mdxrename to docs/guides/avoid-blocking.mdxindex 10cbeeca6..515ee278e 100644--- a/docs/guides/avoid_blocking.mdx+++ b/docs/guides/avoid-blocking.mdx@@ -10,11 +10,11 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import PlaywrightSource from '!!raw-loader!./avoid_blocking_playwright.ts';-import PuppeteerSource from '!!raw-loader!./avoid_blocking_puppeteer';-import PlaywrightFingerprintsOffSource from '!!raw-loader!./avoid_blocking_playwright_fingerprints_off.ts';-import PuppeteerFingerprintsOffSource from '!!raw-loader!./avoid_blocking_puppeteer_fingerprints_off.ts';-import PlaywrightCamoufox from '!!raw-loader!./avoid_blocking_camoufox.ts';+import PlaywrightSource from '!!raw-loader!./avoid-blocking-playwright.ts';+import PuppeteerSource from '!!raw-loader!./avoid-blocking-puppeteer';+import PlaywrightFingerprintsOffSource from '!!raw-loader!./avoid-blocking-playwright-fingerprints-off.ts';+import PuppeteerFingerprintsOffSource from '!!raw-loader!./avoid-blocking-puppeteer-fingerprints-off.ts';+import PlaywrightCamoufox from '!!raw-loader!./avoid-blocking-camoufox.ts';A scraper might get blocked for numerous reasons. Let's narrow it down to the two main ones. The first is a bad or blocked IP address. You can learn about this topic in the [proxy management guide](./proxy-management). The second reason is [browser fingerprints](https://pixelprivacy.com/resources/browser-fingerprinting/) (or signatures), which we will explore more in this guide. Check the [Apify Academy anti-scraping course](https://docs.apify.com/academy/anti-scraping) to gain a deeper theoretical understanding of blocking and learn a few tips and tricks.diff --git a/docs/guides/cheerio_crawler.mdx b/docs/guides/cheerio-crawler.mdxsimilarity index 100%rename from docs/guides/cheerio_crawler.mdxrename to docs/guides/cheerio-crawler.mdxdiff --git a/docs/guides/cookie_modals.mdx b/docs/guides/cookie-modals.mdxsimilarity index 100%rename from docs/guides/cookie_modals.mdxrename to docs/guides/cookie-modals.mdxdiff --git a/docs/guides/custom_statistics_adaptive.ts b/docs/guides/custom-statistics-adaptive.tssimilarity index 100%rename from docs/guides/custom_statistics_adaptive.tsrename to docs/guides/custom-statistics-adaptive.tsdiff --git a/docs/guides/custom_statistics_basic.ts b/docs/guides/custom-statistics-basic.tssimilarity index 100%rename from docs/guides/custom_statistics_basic.tsrename to docs/guides/custom-statistics-basic.tsdiff --git a/docs/guides/custom_statistics_validated.ts b/docs/guides/custom-statistics-validated.tssimilarity index 100%rename from docs/guides/custom_statistics_validated.tsrename to docs/guides/custom-statistics-validated.tsdiff --git a/docs/guides/custom_statistics.mdx b/docs/guides/custom-statistics.mdxsimilarity index 95%rename from docs/guides/custom_statistics.mdxrename to docs/guides/custom-statistics.mdxindex 2b3845a4f..c97b889b2 100644--- a/docs/guides/custom_statistics.mdx+++ b/docs/guides/custom-statistics.mdx@@ -7,9 +7,9 @@ description: Track your own counters alongside the built-in crawler statisticsimport ApiLink from '@site/src/components/ApiLink';import CodeBlock from '@theme/CodeBlock';-import BasicSource from '!!raw-loader!./custom_statistics_basic.ts';-import ValidatedSource from '!!raw-loader!./custom_statistics_validated.ts';-import AdaptiveSource from '!!raw-loader!./custom_statistics_adaptive.ts';+import BasicSource from '!!raw-loader!./custom-statistics-basic.ts';+import ValidatedSource from '!!raw-loader!./custom-statistics-validated.ts';+import AdaptiveSource from '!!raw-loader!./custom-statistics-adaptive.ts';Every crawler collects run statistics — requests finished, failures, retries, durations — into a <ApiLink to="basic-crawler/class/Statistics">`Statistics`</ApiLink> instance exposed as `crawler.statistics`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.diff --git a/docs/guides/docker_images.mdx b/docs/guides/docker-images.mdxsimilarity index 100%rename from docs/guides/docker_images.mdxrename to docs/guides/docker-images.mdxdiff --git a/docs/guides/got_scraping.mdx b/docs/guides/got-scraping.mdxsimilarity index 100%rename from docs/guides/got_scraping.mdxrename to docs/guides/got-scraping.mdxdiff --git a/docs/guides/proxy_management_inspection_cheerio.ts b/docs/guides/proxy-management-inspection-cheerio.tssimilarity index 100%rename from docs/guides/proxy_management_inspection_cheerio.tsrename to docs/guides/proxy-management-inspection-cheerio.tsdiff --git a/docs/guides/proxy_management_inspection_http.ts b/docs/guides/proxy-management-inspection-http.tssimilarity index 100%rename from docs/guides/proxy_management_inspection_http.tsrename to docs/guides/proxy-management-inspection-http.tsdiff --git a/docs/guides/proxy_management_inspection_playwright.ts b/docs/guides/proxy-management-inspection-playwright.tssimilarity index 100%rename from docs/guides/proxy_management_inspection_playwright.tsrename to docs/guides/proxy-management-inspection-playwright.tsdiff --git a/docs/guides/proxy_management_inspection_puppeteer.ts b/docs/guides/proxy-management-inspection-puppeteer.tssimilarity index 100%rename from docs/guides/proxy_management_inspection_puppeteer.tsrename to docs/guides/proxy-management-inspection-puppeteer.tsdiff --git a/docs/guides/proxy_management_integration_cheerio.ts b/docs/guides/proxy-management-integration-cheerio.tssimilarity index 100%rename from docs/guides/proxy_management_integration_cheerio.tsrename to docs/guides/proxy-management-integration-cheerio.tsdiff --git a/docs/guides/proxy_management_integration_http.ts b/docs/guides/proxy-management-integration-http.tssimilarity index 100%rename from docs/guides/proxy_management_integration_http.tsrename to docs/guides/proxy-management-integration-http.tsdiff --git a/docs/guides/proxy_management_integration_playwright.ts b/docs/guides/proxy-management-integration-playwright.tssimilarity index 100%rename from docs/guides/proxy_management_integration_playwright.tsrename to docs/guides/proxy-management-integration-playwright.tsdiff --git a/docs/guides/proxy_management_integration_puppeteer.ts b/docs/guides/proxy-management-integration-puppeteer.tssimilarity index 100%rename from docs/guides/proxy_management_integration_puppeteer.tsrename to docs/guides/proxy-management-integration-puppeteer.tsdiff --git a/docs/guides/proxy_management_session_cheerio.ts b/docs/guides/proxy-management-session-cheerio.tssimilarity index 100%rename from docs/guides/proxy_management_session_cheerio.tsrename to docs/guides/proxy-management-session-cheerio.tsdiff --git a/docs/guides/proxy_management_session_http.ts b/docs/guides/proxy-management-session-http.tssimilarity index 100%rename from docs/guides/proxy_management_session_http.tsrename to docs/guides/proxy-management-session-http.tsdiff --git a/docs/guides/proxy_management_session_playwright.ts b/docs/guides/proxy-management-session-playwright.tssimilarity index 100%rename from docs/guides/proxy_management_session_playwright.tsrename to docs/guides/proxy-management-session-playwright.tsdiff --git a/docs/guides/proxy_management_session_puppeteer.ts b/docs/guides/proxy-management-session-puppeteer.tssimilarity index 100%rename from docs/guides/proxy_management_session_puppeteer.tsrename to docs/guides/proxy-management-session-puppeteer.tsdiff --git a/docs/guides/proxy_management_session_standalone.ts b/docs/guides/proxy-management-session-standalone.tssimilarity index 100%rename from docs/guides/proxy_management_session_standalone.tsrename to docs/guides/proxy-management-session-standalone.tsdiff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy-management.mdxsimilarity index 85%rename from docs/guides/proxy_management.mdxrename to docs/guides/proxy-management.mdxindex 65d07094b..873dba333 100644--- a/docs/guides/proxy_management.mdx+++ b/docs/guides/proxy-management.mdx@@ -10,19 +10,19 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import HttpSource from '!!raw-loader!./proxy_management_integration_http.ts';-import CheerioSource from '!!raw-loader!./proxy_management_integration_cheerio.ts';-import PlaywrightSource from '!!raw-loader!./proxy_management_integration_playwright.ts';-import PuppeteerSource from '!!raw-loader!./proxy_management_integration_puppeteer.ts';-import SessionStandaloneSource from '!!raw-loader!./proxy_management_session_standalone.ts';-import SessionHttpSource from '!!raw-loader!./proxy_management_session_http.ts';-import SessionCheerioSource from '!!raw-loader!./proxy_management_session_cheerio.ts';-import SessionPlaywrightSource from '!!raw-loader!./proxy_management_session_playwright.ts';-import SessionPuppeteerSource from '!!raw-loader!./proxy_management_session_puppeteer.ts';-import InspectionHttpSource from '!!raw-loader!./proxy_management_inspection_http.ts';-import InspectionCheerioSource from '!!raw-loader!./proxy_management_inspection_cheerio.ts';-import InspectionPlaywrightSource from '!!raw-loader!./proxy_management_inspection_playwright.ts';-import InspectionPuppeteerSource from '!!raw-loader!./proxy_management_inspection_puppeteer.ts';+import HttpSource from '!!raw-loader!./proxy-management-integration-http.ts';+import CheerioSource from '!!raw-loader!./proxy-management-integration-cheerio.ts';+import PlaywrightSource from '!!raw-loader!./proxy-management-integration-playwright.ts';+import PuppeteerSource from '!!raw-loader!./proxy-management-integration-puppeteer.ts';+import SessionStandaloneSource from '!!raw-loader!./proxy-management-session-standalone.ts';+import SessionHttpSource from '!!raw-loader!./proxy-management-session-http.ts';+import SessionCheerioSource from '!!raw-loader!./proxy-management-session-cheerio.ts';+import SessionPlaywrightSource from '!!raw-loader!./proxy-management-session-playwright.ts';+import SessionPuppeteerSource from '!!raw-loader!./proxy-management-session-puppeteer.ts';+import InspectionHttpSource from '!!raw-loader!./proxy-management-inspection-http.ts';+import InspectionCheerioSource from '!!raw-loader!./proxy-management-inspection-cheerio.ts';+import InspectionPlaywrightSource from '!!raw-loader!./proxy-management-inspection-playwright.ts';+import InspectionPuppeteerSource from '!!raw-loader!./proxy-management-inspection-puppeteer.ts';[IP address blocking](https://en.wikipedia.org/wiki/IP_address_blocking) is one of the oldestand most effective ways of preventing access to a website. It is therefore paramount fordiff --git a/docs/guides/public_api.mdx b/docs/guides/public-api.mdxsimilarity index 100%rename from docs/guides/public_api.mdxrename to docs/guides/public-api.mdxdiff --git a/docs/guides/remote_browser_config.ts b/docs/guides/remote-browser-config.tssimilarity index 100%rename from docs/guides/remote_browser_config.tsrename to docs/guides/remote-browser-config.tsdiff --git a/docs/guides/remote_browser_provider.ts b/docs/guides/remote-browser-provider.tssimilarity index 100%rename from docs/guides/remote_browser_provider.tsrename to docs/guides/remote-browser-provider.tsdiff --git a/docs/guides/remote_browser_puppeteer.ts b/docs/guides/remote-browser-puppeteer.tssimilarity index 100%rename from docs/guides/remote_browser_puppeteer.tsrename to docs/guides/remote-browser-puppeteer.tsdiff --git a/docs/guides/remote_browser.mdx b/docs/guides/remote-browser.mdxsimilarity index 96%rename from docs/guides/remote_browser.mdxrename to docs/guides/remote-browser.mdxindex bb0c5a159..e3e7b7446 100644--- a/docs/guides/remote_browser.mdx+++ b/docs/guides/remote-browser.mdx@@ -8,9 +8,9 @@ description: Connect Crawlee crawlers to remote browser services like Browserbasimport ApiLink from '@site/src/components/ApiLink';import CodeBlock from '@theme/CodeBlock';-import RemoteBrowserConfigSource from '!!raw-loader!./remote_browser_config.ts';-import RemoteBrowserProviderSource from '!!raw-loader!./remote_browser_provider.ts';-import RemoteBrowserPuppeteerSource from '!!raw-loader!./remote_browser_puppeteer.ts';+import RemoteBrowserConfigSource from '!!raw-loader!./remote-browser-config.ts';+import RemoteBrowserProviderSource from '!!raw-loader!./remote-browser-provider.ts';+import RemoteBrowserPuppeteerSource from '!!raw-loader!./remote-browser-puppeteer.ts';Instead of launching a local browser, Crawlee can connect to a remote browser service like [Browserbase](https://browserbase.com/), [Browserless](https://browserless.io/), [Steel](https://steel.dev/), or any service that exposes a WebSocket/CDP endpoint. The crawler manages session rotation and the request lifecycle the same way it does locally — only the browser itself runs elsewhere.diff --git a/docs/guides/request_loaders_rl_basic.ts b/docs/guides/request-loaders-rl-basic.tssimilarity index 100%rename from docs/guides/request_loaders_rl_basic.tsrename to docs/guides/request-loaders-rl-basic.tsdiff --git a/docs/guides/request_loaders_rl_tandem_explicit.ts b/docs/guides/request-loaders-rl-tandem-explicit.tssimilarity index 100%rename from docs/guides/request_loaders_rl_tandem_explicit.tsrename to docs/guides/request-loaders-rl-tandem-explicit.tsdiff --git a/docs/guides/request_loaders_rl_tandem_helper.ts b/docs/guides/request-loaders-rl-tandem-helper.tssimilarity index 100%rename from docs/guides/request_loaders_rl_tandem_helper.tsrename to docs/guides/request-loaders-rl-tandem-helper.tsdiff --git a/docs/guides/request_loaders_sitemap_basic.ts b/docs/guides/request-loaders-sitemap-basic.tssimilarity index 100%rename from docs/guides/request_loaders_sitemap_basic.tsrename to docs/guides/request-loaders-sitemap-basic.tsdiff --git a/docs/guides/request_loaders_sitemap_tandem_explicit.ts b/docs/guides/request-loaders-sitemap-tandem-explicit.tssimilarity index 100%rename from docs/guides/request_loaders_sitemap_tandem_explicit.tsrename to docs/guides/request-loaders-sitemap-tandem-explicit.tsdiff --git a/docs/guides/request_loaders_sitemap_tandem_helper.ts b/docs/guides/request-loaders-sitemap-tandem-helper.tssimilarity index 100%rename from docs/guides/request_loaders_sitemap_tandem_helper.tsrename to docs/guides/request-loaders-sitemap-tandem-helper.tsdiff --git a/docs/guides/request_loaders.mdx b/docs/guides/request-loaders.mdxsimilarity index 97%rename from docs/guides/request_loaders.mdxrename to docs/guides/request-loaders.mdxindex fdd8e8abe..ba13ecd30 100644--- a/docs/guides/request_loaders.mdx+++ b/docs/guides/request-loaders.mdx@@ -10,12 +10,12 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import RlBasicSource from '!!raw-loader!./request_loaders_rl_basic.ts';-import SitemapBasicSource from '!!raw-loader!./request_loaders_sitemap_basic.ts';-import RlTandemExplicitSource from '!!raw-loader!./request_loaders_rl_tandem_explicit.ts';-import RlTandemHelperSource from '!!raw-loader!./request_loaders_rl_tandem_helper.ts';-import SitemapTandemExplicitSource from '!!raw-loader!./request_loaders_sitemap_tandem_explicit.ts';-import SitemapTandemHelperSource from '!!raw-loader!./request_loaders_sitemap_tandem_helper.ts';+import RlBasicSource from '!!raw-loader!./request-loaders-rl-basic.ts';+import SitemapBasicSource from '!!raw-loader!./request-loaders-sitemap-basic.ts';+import RlTandemExplicitSource from '!!raw-loader!./request-loaders-rl-tandem-explicit.ts';+import RlTandemHelperSource from '!!raw-loader!./request-loaders-rl-tandem-helper.ts';+import SitemapTandemExplicitSource from '!!raw-loader!./request-loaders-sitemap-tandem-explicit.ts';+import SitemapTandemHelperSource from '!!raw-loader!./request-loaders-sitemap-tandem-helper.ts';Request loaders extend the functionality of the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, providing additional tools for managing URLs and requests. If you are new to Crawlee and unfamiliar with the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, consider starting with the [Request storage](./request-storage) guide first. Request loaders define how requests are fetched and stored, enabling various use cases such as reading URLs from a static list, a sitemap, an external API, or combining multiple sources together.diff --git a/docs/guides/request_storage_queue_basic.ts b/docs/guides/request-storage-queue-basic.tssimilarity index 100%rename from docs/guides/request_storage_queue_basic.tsrename to docs/guides/request-storage-queue-basic.tsdiff --git a/docs/guides/request_storage_queue_crawler_explicit.ts b/docs/guides/request-storage-queue-crawler-explicit.tssimilarity index 100%rename from docs/guides/request_storage_queue_crawler_explicit.tsrename to docs/guides/request-storage-queue-crawler-explicit.tsdiff --git a/docs/guides/request_storage_queue_crawler.ts b/docs/guides/request-storage-queue-crawler.tssimilarity index 100%rename from docs/guides/request_storage_queue_crawler.tsrename to docs/guides/request-storage-queue-crawler.tsdiff --git a/docs/guides/request_storage_queue_only.ts b/docs/guides/request-storage-queue-only.tssimilarity index 100%rename from docs/guides/request_storage_queue_only.tsrename to docs/guides/request-storage-queue-only.tsdiff --git a/docs/guides/request_storage.mdx b/docs/guides/request-storage.mdxsimilarity index 95%rename from docs/guides/request_storage.mdxrename to docs/guides/request-storage.mdxindex 803450c1b..8a8fbcd08 100644--- a/docs/guides/request_storage.mdx+++ b/docs/guides/request-storage.mdx@@ -10,11 +10,11 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import BasicOperationsSource from '!!raw-loader!./request_storage_queue_basic.ts';-import CrawlerExplicitSource from '!!raw-loader!./request_storage_queue_crawler_explicit.ts';-import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';+import BasicOperationsSource from '!!raw-loader!./request-storage-queue-basic.ts';+import CrawlerExplicitSource from '!!raw-loader!./request-storage-queue-crawler-explicit.ts';+import CrawlerSource from '!!raw-loader!./request-storage-queue-crawler.ts';-import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';+import RequestQueueAddRequestsSource from '!!raw-loader!./request-storage-queue-only.ts';Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.diff --git a/docs/guides/result_storage.mdx b/docs/guides/result-storage.mdxsimilarity index 100%rename from docs/guides/result_storage.mdxrename to docs/guides/result-storage.mdxdiff --git a/docs/guides/scaling_crawlers_concurrencySystem.ts b/docs/guides/scaling-crawlers-concurrency-system.tssimilarity index 100%rename from docs/guides/scaling_crawlers_concurrencySystem.tsrename to docs/guides/scaling-crawlers-concurrency-system.tsdiff --git a/docs/guides/scaling_crawlers_customConcurrencySystem.ts b/docs/guides/scaling-crawlers-custom-concurrency-system.tssimilarity index 100%rename from docs/guides/scaling_crawlers_customConcurrencySystem.tsrename to docs/guides/scaling-crawlers-custom-concurrency-system.tsdiff --git a/docs/guides/scaling_crawlers_customLoadSignal.ts b/docs/guides/scaling-crawlers-custom-load-signal.tssimilarity index 100%rename from docs/guides/scaling_crawlers_customLoadSignal.tsrename to docs/guides/scaling-crawlers-custom-load-signal.tsdiff --git a/docs/guides/scaling_crawlers_loadSignalsOff.ts b/docs/guides/scaling-crawlers-load-signals-off.tssimilarity index 100%rename from docs/guides/scaling_crawlers_loadSignalsOff.tsrename to docs/guides/scaling-crawlers-load-signals-off.tsdiff --git a/docs/guides/scaling_crawlers_maxRequestsPerMinute.ts b/docs/guides/scaling-crawlers-max-requests-per-minute.tssimilarity index 100%rename from docs/guides/scaling_crawlers_maxRequestsPerMinute.tsrename to docs/guides/scaling-crawlers-max-requests-per-minute.tsdiff --git a/docs/guides/scaling_crawlers_minMaxConcurrency.ts b/docs/guides/scaling-crawlers-min-max-concurrency.tssimilarity index 100%rename from docs/guides/scaling_crawlers_minMaxConcurrency.tsrename to docs/guides/scaling-crawlers-min-max-concurrency.tsdiff --git a/docs/guides/scaling_crawlers_wrapLoadSignal.ts b/docs/guides/scaling-crawlers-wrap-load-signal.tssimilarity index 100%rename from docs/guides/scaling_crawlers_wrapLoadSignal.tsrename to docs/guides/scaling-crawlers-wrap-load-signal.tsdiff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling-crawlers.mdxsimilarity index 94%rename from docs/guides/scaling_crawlers.mdxrename to docs/guides/scaling-crawlers.mdxindex ecc9fabe5..d551ae039 100644--- a/docs/guides/scaling_crawlers.mdx+++ b/docs/guides/scaling-crawlers.mdx@@ -8,13 +8,13 @@ import ApiLink from '@site/src/components/ApiLink';import CodeBlock from '@theme/CodeBlock';-import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';-import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';-import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrencySystem.ts';-import LoadSignalsOffSource from '!!raw-loader!./scaling_crawlers_loadSignalsOff.ts';-import CustomLoadSignalSource from '!!raw-loader!./scaling_crawlers_customLoadSignal.ts';-import WrapLoadSignalSource from '!!raw-loader!./scaling_crawlers_wrapLoadSignal.ts';-import CustomConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_customConcurrencySystem.ts';+import MinMaxConcurrencySource from '!!raw-loader!./scaling-crawlers-min-max-concurrency.ts';+import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling-crawlers-max-requests-per-minute.ts';+import ConcurrencySystemSource from '!!raw-loader!./scaling-crawlers-concurrency-system.ts';+import LoadSignalsOffSource from '!!raw-loader!./scaling-crawlers-load-signals-off.ts';+import CustomLoadSignalSource from '!!raw-loader!./scaling-crawlers-custom-load-signal.ts';+import WrapLoadSignalSource from '!!raw-loader!./scaling-crawlers-wrap-load-signal.ts';+import CustomConcurrencySystemSource from '!!raw-loader!./scaling-crawlers-custom-concurrency-system.ts';As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.diff --git a/docs/guides/session_management_basic.ts b/docs/guides/session-management-basic.tssimilarity index 100%rename from docs/guides/session_management_basic.tsrename to docs/guides/session-management-basic.tsdiff --git a/docs/guides/session_management_cheerio.ts b/docs/guides/session-management-cheerio.tssimilarity index 100%rename from docs/guides/session_management_cheerio.tsrename to docs/guides/session-management-cheerio.tsdiff --git a/docs/guides/session_management_http.ts b/docs/guides/session-management-http.tssimilarity index 100%rename from docs/guides/session_management_http.tsrename to docs/guides/session-management-http.tsdiff --git a/docs/guides/session_management_playwright.ts b/docs/guides/session-management-playwright.tssimilarity index 100%rename from docs/guides/session_management_playwright.tsrename to docs/guides/session-management-playwright.tsdiff --git a/docs/guides/session_management_puppeteer.ts b/docs/guides/session-management-puppeteer.tssimilarity index 100%rename from docs/guides/session_management_puppeteer.tsrename to docs/guides/session-management-puppeteer.tsdiff --git a/docs/guides/session_management_standalone.ts b/docs/guides/session-management-standalone.tssimilarity index 100%rename from docs/guides/session_management_standalone.tsrename to docs/guides/session-management-standalone.tsdiff --git a/docs/guides/session_management.mdx b/docs/guides/session-management.mdxsimilarity index 97%rename from docs/guides/session_management.mdxrename to docs/guides/session-management.mdxindex eac1cb527..210b9808c 100644--- a/docs/guides/session_management.mdx+++ b/docs/guides/session-management.mdx@@ -10,12 +10,12 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import BasicSource from '!!raw-loader!./session_management_basic.ts';-import HttpSource from '!!raw-loader!./session_management_http.ts';-import CheerioSource from '!!raw-loader!./session_management_cheerio.ts';-import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';-import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';-import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';+import BasicSource from '!!raw-loader!./session-management-basic.ts';+import HttpSource from '!!raw-loader!./session-management-http.ts';+import CheerioSource from '!!raw-loader!./session-management-cheerio.ts';+import PlaywrightSource from '!!raw-loader!./session-management-playwright.ts';+import PuppeteerSource from '!!raw-loader!./session-management-puppeteer.ts';+import StandaloneSource from '!!raw-loader!./session-management-standalone.ts';​<ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single <ApiLink to="basic-crawler/class/Session">`Session`</ApiLink> bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).diff --git a/docs/guides/stagehand_crawler_basic.ts b/docs/guides/stagehand-crawler-basic.tssimilarity index 100%rename from docs/guides/stagehand_crawler_basic.tsrename to docs/guides/stagehand-crawler-basic.tsdiff --git a/docs/guides/stagehand_crawler_combined.ts b/docs/guides/stagehand-crawler-combined.tssimilarity index 100%rename from docs/guides/stagehand_crawler_combined.tsrename to docs/guides/stagehand-crawler-combined.tsdiff --git a/docs/guides/stagehand_crawler_extract.ts b/docs/guides/stagehand-crawler-extract.tssimilarity index 100%rename from docs/guides/stagehand_crawler_extract.tsrename to docs/guides/stagehand-crawler-extract.tsdiff --git a/docs/guides/stagehand_crawler.mdx b/docs/guides/stagehand-crawler.mdxsimilarity index 97%rename from docs/guides/stagehand_crawler.mdxrename to docs/guides/stagehand-crawler.mdxindex fb8f4cfc9..f274c20cf 100644--- a/docs/guides/stagehand_crawler.mdx+++ b/docs/guides/stagehand-crawler.mdx@@ -10,9 +10,9 @@ import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';-import StagehandBasicSource from '!!raw-loader!./stagehand_crawler_basic.ts';-import StagehandExtractSource from '!!raw-loader!./stagehand_crawler_extract.ts';-import StagehandCombinedSource from '!!raw-loader!./stagehand_crawler_combined.ts';+import StagehandBasicSource from '!!raw-loader!./stagehand-crawler-basic.ts';+import StagehandExtractSource from '!!raw-loader!./stagehand-crawler-extract.ts';+import StagehandCombinedSource from '!!raw-loader!./stagehand-crawler-combined.ts';​<ApiLink to="stagehand-crawler/class/StagehandCrawler">`StagehandCrawler`</ApiLink> combines Crawlee's powerful crawling infrastructure with [Stagehand's](https://github.com/browserbase/stagehand) AI-powered browser automation. Instead of writing CSS selectors or XPath queries, you can interact with web pages using natural language instructions.diff --git a/docs/guides/trace_and_monitor_basic.ts b/docs/guides/trace-and-monitor-basic.tssimilarity index 100%rename from docs/guides/trace_and_monitor_basic.tsrename to docs/guides/trace-and-monitor-basic.tsdiff --git a/docs/guides/trace-and-monitor-crawlers.mdx b/docs/guides/trace-and-monitor-crawlers.mdxindex 1b1b6fa92..cfacfb82c 100644--- a/docs/guides/trace-and-monitor-crawlers.mdx+++ b/docs/guides/trace-and-monitor-crawlers.mdx@@ -6,11 +6,11 @@ description: How to use OpenTelemetry to trace and monitor your crawlersimport CodeBlock from '@theme/CodeBlock';-import RegisterHookSource from '!!raw-loader!./trace_and_monitor_register_hook.ts';-import SetupSource from '!!raw-loader!./trace_and_monitor_setup.ts';-import BasicExampleSource from '!!raw-loader!./trace_and_monitor_basic.ts';-import WrapWithSpanSource from '!!raw-loader!./trace_and_monitor_wrap_with_span.ts';-import CustomInstrumentationSource from '!!raw-loader!./trace_and_monitor_custom.ts';+import RegisterHookSource from '!!raw-loader!./trace-and-monitor-register-hook.ts';+import SetupSource from '!!raw-loader!./trace-and-monitor-setup.ts';+import BasicExampleSource from '!!raw-loader!./trace-and-monitor-basic.ts';+import WrapWithSpanSource from '!!raw-loader!./trace-and-monitor-wrap-with-span.ts';+import CustomInstrumentationSource from '!!raw-loader!./trace-and-monitor-custom.ts';[OpenTelemetry](https://opentelemetry.io/) is a collection of APIs, SDKs, and tools to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software's performance and behavior. You can learn more about its basic concepts in the [OpenTelemetry documentation](https://opentelemetry.io/docs/concepts/).@@ -98,9 +98,9 @@ starts, and only then is the crawler loaded and patched.The examples on this page live in the Crawlee repository, so to run this one from a checkout, from the repository root:```bash-pnpm exec tsx --import ./docs/guides/trace_and_monitor_register_hook.ts \- --import ./docs/guides/trace_and_monitor_setup.ts \- ./docs/guides/trace_and_monitor_basic.ts+pnpm exec tsx --import ./docs/guides/trace-and-monitor-register-hook.ts \+ --import ./docs/guides/trace-and-monitor-setup.ts \+ ./docs/guides/trace-and-monitor-basic.ts```## Troubleshoot instrumentation setupdiff --git a/docs/guides/trace_and_monitor_custom.ts b/docs/guides/trace-and-monitor-custom.tssimilarity index 100%rename from docs/guides/trace_and_monitor_custom.tsrename to docs/guides/trace-and-monitor-custom.tsdiff --git a/docs/guides/trace_and_monitor_register_hook.ts b/docs/guides/trace-and-monitor-register-hook.tssimilarity index 100%rename from docs/guides/trace_and_monitor_register_hook.tsrename to docs/guides/trace-and-monitor-register-hook.tsdiff --git a/docs/guides/trace_and_monitor_setup.ts b/docs/guides/trace-and-monitor-setup.tssimilarity index 100%rename from docs/guides/trace_and_monitor_setup.tsrename to docs/guides/trace-and-monitor-setup.tsdiff --git a/docs/guides/trace_and_monitor_wrap_with_span.ts b/docs/guides/trace-and-monitor-wrap-with-span.tssimilarity index 100%rename from docs/guides/trace_and_monitor_wrap_with_span.tsrename to docs/guides/trace-and-monitor-wrap-with-span.tsdiff --git a/docs/guides/typescript_project.mdx b/docs/guides/typescript-project.mdxsimilarity index 100%rename from docs/guides/typescript_project.mdxrename to docs/guides/typescript-project.mdxdiff --git a/docs/public-api/README.md b/docs/public-api/README.mdindex 6368024aa..db9cdd7ae 100644--- a/docs/public-api/README.md+++ b/docs/public-api/README.md@@ -1,6 +1,6 @@# Public API surface maps-Each `*.api.md` file here is a generated map of the public, type-level interface of one `@crawlee/*` package. Being in a report is the backwards-compatibility promise; being absent means no promise, not that the symbol is unreachable. See the [Public API guide](../guides/public_api.mdx) for what that means for users.+Each `*.api.md` file here is a generated map of the public, type-level interface of one `@crawlee/*` package. Being in a report is the backwards-compatibility promise; being absent means no promise, not that the symbol is unreachable. See the [Public API guide](../guides/public-api.mdx) for what that means for users.- **Untagged** members are promised. The codebase does not use explicit `@public` tags.- **`@internal`** (and the legacy `@ignore`) members are trimmed from the report but stay exported and present in the `.d.ts`. Use `#private` / `private` when something should be unreachable, `@internal` when it should be reachable but unsupported.diff --git a/docs/quick-start/headful_playwright.ts b/docs/quick-start/headful-playwright.tssimilarity index 100%rename from docs/quick-start/headful_playwright.tsrename to docs/quick-start/headful-playwright.tsdiff --git a/docs/quick-start/headful_puppeteer.ts b/docs/quick-start/headful-puppeteer.tssimilarity index 100%rename from docs/quick-start/headful_puppeteer.tsrename to docs/quick-start/headful-puppeteer.tsdiff --git a/docs/quick-start/index.mdx b/docs/quick-start/index.mdxindex 63a9f6178..2d68c75f8 100644--- a/docs/quick-start/index.mdx+++ b/docs/quick-start/index.mdx@@ -13,11 +13,11 @@ import TabItem from '@theme/TabItem';import CodeBlock from '@theme/CodeBlock';import ThemedImage from '@theme/ThemedImage';-import CheerioSource from '!!raw-loader!roa-loader!./quick_start_cheerio.ts';-import PlaywrightSource from '!!raw-loader!roa-loader!./quick_start_playwright.ts';-import PlaywrightHeadful from '!!raw-loader!roa-loader!./headful_playwright.ts';-import PuppeteerSource from '!!raw-loader!roa-loader!./quick_start_puppeteer.ts';-import PuppeteerHeadful from '!!raw-loader!roa-loader!./headful_puppeteer.ts';+import CheerioSource from '!!raw-loader!roa-loader!./quick-start-cheerio.ts';+import PlaywrightSource from '!!raw-loader!roa-loader!./quick-start-playwright.ts';+import PlaywrightHeadful from '!!raw-loader!roa-loader!./headful-playwright.ts';+import PuppeteerSource from '!!raw-loader!roa-loader!./quick-start-puppeteer.ts';+import PuppeteerHeadful from '!!raw-loader!roa-loader!./headful-puppeteer.ts';import CheerioLog from '!!raw-loader!./quick_start_cheerio.txt';diff --git a/docs/quick-start/quick_start_cheerio.ts b/docs/quick-start/quick-start-cheerio.tssimilarity index 100%rename from docs/quick-start/quick_start_cheerio.tsrename to docs/quick-start/quick-start-cheerio.tsdiff --git a/docs/quick-start/quick_start_playwright.ts b/docs/quick-start/quick-start-playwright.tssimilarity index 100%rename from docs/quick-start/quick_start_playwright.tsrename to docs/quick-start/quick-start-playwright.tsdiff --git a/docs/quick-start/quick_start_puppeteer.ts b/docs/quick-start/quick-start-puppeteer.tssimilarity index 100%rename from docs/quick-start/quick_start_puppeteer.tsrename to docs/quick-start/quick-start-puppeteer.tsdiff --git a/docs/upgrading/upgrading_v1.md b/docs/upgrading/upgrading-v1.mdsimilarity index 100%rename from docs/upgrading/upgrading_v1.mdrename to docs/upgrading/upgrading-v1.mddiff --git a/docs/upgrading/upgrading_v2.md b/docs/upgrading/upgrading-v2.mdsimilarity index 100%rename from docs/upgrading/upgrading_v2.mdrename to docs/upgrading/upgrading-v2.mddiff --git a/docs/upgrading/upgrading_v3.md b/docs/upgrading/upgrading-v3.mdsimilarity index 99%rename from docs/upgrading/upgrading_v3.mdrename to docs/upgrading/upgrading-v3.mdindex d61fc560a..94a538630 100644--- a/docs/upgrading/upgrading_v3.md+++ b/docs/upgrading/upgrading-v3.md@@ -256,7 +256,7 @@ const crawler = new BasicCrawler({### How to use `sendRequest()`?-See [the Got Scraping guide](../guides/got_scraping.mdx).+See [the Got Scraping guide](../guides/got-scraping.mdx).### Removed optionsdiff --git a/docs/upgrading/upgrading_v4/_category_.json b/docs/upgrading/upgrading-v4/_category_.jsonsimilarity index 100%rename from docs/upgrading/upgrading_v4/_category_.jsonrename to docs/upgrading/upgrading-v4/_category_.jsondiff --git a/docs/upgrading/upgrading_v4/autoscaling.md b/docs/upgrading/upgrading-v4/autoscaling.mdsimilarity index 97%rename from docs/upgrading/upgrading_v4/autoscaling.mdrename to docs/upgrading/upgrading-v4/autoscaling.mdindex d604d7184..4f9fa05cb 100644--- a/docs/upgrading/upgrading_v4/autoscaling.md+++ b/docs/upgrading/upgrading-v4/autoscaling.md@@ -6,7 +6,7 @@ sidebar_position: 8slug: /upgrading/upgrading-to-v4/autoscaling----The `minConcurrency` / `maxConcurrency` / `maxRequestsPerMinute` crawler options work as before. This section matters when you used `autoscaledPoolOptions`, drove an `AutoscaledPool` directly, or configured snapshotting and system status. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+The `minConcurrency` / `maxConcurrency` / `maxRequestsPerMinute` crawler options work as before. This section matters when you used `autoscaledPoolOptions`, drove an `AutoscaledPool` directly, or configured snapshotting and system status. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Autoscaling moved to `ConcurrencySystem`@@ -39,7 +39,7 @@ try {}```-On Node.js 24, `await using` replaces the `try`/`finally` — see [collaborators you own are disposable](./upgrading_v4.md#collaborators-you-own-are-disposable).+On Node.js 24, `await using` replaces the `try`/`finally` — see [collaborators you own are disposable](./upgrading-v4.md#collaborators-you-own-are-disposable).### `AutoscaledPool` is no longer public API@@ -219,11 +219,11 @@ new ConcurrencySystem({});```-The signal that watches storage rate-limit errors follows the [`StorageClient` → `StorageBackend` rename](./storage_backends.md#storagebackend-interface-simplified): the option is `loadSignals.storageBackend`, the class `StorageBackendLoadSignal`, and its verdict is reported as `SystemInfo.storageBackendInfo`. The type of that verdict, `ClientInfo`, is now `LoadSignalInfo` — it backs every signal's entry, not just this one.+The signal that watches storage rate-limit errors follows the [`StorageClient` → `StorageBackend` rename](./storage-backends.md#storagebackend-interface-simplified): the option is `loadSignals.storageBackend`, the class `StorageBackendLoadSignal`, and its verdict is reported as `SystemInfo.storageBackendInfo`. The type of that verdict, `ClientInfo`, is now `LoadSignalInfo` — it backs every signal's entry, not just this one.In detail: the four `max*OverloadedRatio` options of `SystemStatusOptions` were **removed** (each signal now owns its overload ratio, set in its own bag), custom signals moved from `systemStatusOptions.loadSignals` to `loadSignals.custom`, and the two evaluation windows — `snapshotHistorySecs` (autoscaling) and `currentHistorySecs` (task gating) — are plain options on `ConcurrencySystemOptions`, since they apply to every signal alike rather than to any one of them.-Two related capabilities are new, and covered in the [scaling guide](../../guides/scaling_crawlers.mdx#load-signals): a built-in signal can be switched **off** with `false`, and each built-in is also a public class (`MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `StorageBackendLoadSignal`) you can construct to wrap or adapt.+Two related capabilities are new, and covered in the [scaling guide](../../guides/scaling-crawlers.mdx#load-signals): a built-in signal can be switched **off** with `false`, and each built-in is also a public class (`MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `StorageBackendLoadSignal`) you can construct to wrap or adapt.A duplicate signal name now **throws**, where naming a custom signal after a built-in (`memInfo`, `eventLoopInfo`, `cpuInfo`, `storageBackendInfo`) used to look like an override but never was one: the built-in kept running and kept holding concurrency down, while your signal only overwrote its field in the reported `SystemInfo`. To take a built-in's place, switch it off:@@ -243,4 +243,4 @@ new ConcurrencySystem({Overload is evaluated over two windows: `currentHistorySecs` (default 5s) gates whether another task may start, and `snapshotHistorySecs` (default 30s) drives autoscaling. The long one used to be implicit — `getHistoricalStatus()` asked each signal for *everything it had retained*, so a custom signal keeping five minutes of snapshots silently made autoscaling reason over five minutes of history for that resource while the built-ins used 30 seconds. Both are now requested explicitly from every signal, and `LoadSignal.start()` receives the wider of the two as `maxSampleWindowMillis` so retention can be sized to match.-You are affected if a custom signal retains **more** history than `snapshotHistorySecs` (its stale snapshots no longer influence scaling), or if its `getSample()` ignores the `sampleDurationMillis` argument, in which case it still contributes everything it has. On the API side, `snapshotHistoryMillis` was removed from the per-signal option types and `SnapshotStore` lost both its constructor argument and the `fromInterval`/`fromEvent` factories; call `useSampleWindow(maxSampleWindowMillis)` and `clear()` from your `start()` instead, as a store that is never given a window retains everything. The [scaling guide](../../guides/scaling_crawlers.mdx#load-signals) has a worked example.+You are affected if a custom signal retains **more** history than `snapshotHistorySecs` (its stale snapshots no longer influence scaling), or if its `getSample()` ignores the `sampleDurationMillis` argument, in which case it still contributes everything it has. On the API side, `snapshotHistoryMillis` was removed from the per-signal option types and `SnapshotStore` lost both its constructor argument and the `fromInterval`/`fromEvent` factories; call `useSampleWindow(maxSampleWindowMillis)` and `clear()` from your `start()` instead, as a store that is never given a window retains everything. The [scaling guide](../../guides/scaling-crawlers.mdx#load-signals) has a worked example.diff --git a/docs/upgrading/upgrading_v4/browser_management.md b/docs/upgrading/upgrading-v4/browser-management.mdsimilarity index 99%rename from docs/upgrading/upgrading_v4/browser_management.mdrename to docs/upgrading/upgrading-v4/browser-management.mdindex da6c32ada..58f1c23fc 100644--- a/docs/upgrading/upgrading_v4/browser_management.md+++ b/docs/upgrading/upgrading-v4/browser-management.md@@ -6,7 +6,7 @@ sidebar_position: 3slug: /upgrading/upgrading-to-v4/browser-management----Applies when you construct a `BrowserPool` yourself, reach for `browserController`, or tune `AdaptivePlaywrightCrawler` internals. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you construct a `BrowserPool` yourself, reach for `browserController`, or tune `AdaptivePlaywrightCrawler` internals. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Custom `BrowserPool` implementations via the `IBrowserPool` interfacediff --git a/docs/upgrading/upgrading_v4/http_clients.md b/docs/upgrading/upgrading-v4/http-clients.mdsimilarity index 99%rename from docs/upgrading/upgrading_v4/http_clients.mdrename to docs/upgrading/upgrading-v4/http-clients.mdindex f197bc6f1..4444b2656 100644--- a/docs/upgrading/upgrading_v4/http_clients.md+++ b/docs/upgrading/upgrading-v4/http-clients.md@@ -6,7 +6,7 @@ sidebar_position: 5slug: /upgrading/upgrading-to-v4/http-clients----Applies when you implemented `BaseHttpClient` yourself, or imported `gotScraping` from `@crawlee/utils`. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you implemented `BaseHttpClient` yourself, or imported `gotScraping` from `@crawlee/utils`. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## HTTP client packages and `BaseHttpClient` reshapeddiff --git a/docs/upgrading/upgrading_v4/internals.md b/docs/upgrading/upgrading-v4/internals.mdsimilarity index 99%rename from docs/upgrading/upgrading_v4/internals.mdrename to docs/upgrading/upgrading-v4/internals.mdindex f218e85c5..cc92b55b9 100644--- a/docs/upgrading/upgrading_v4/internals.md+++ b/docs/upgrading/upgrading-v4/internals.md@@ -6,7 +6,7 @@ sidebar_position: 1slug: /upgrading/upgrading-to-v4/internals----Skip this section unless you subclass Crawlee classes, override `protected` members, or read fields that were never documented. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Skip this section unless you subclass Crawlee classes, override `protected` members, or read fields that were never documented. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Underscore prefix is removed from many protected and private methods@@ -65,7 +65,7 @@ The change spans, among others:- **`BrowserCrawler`** — `navigationTimeoutMillis`, `preNavigationHooks`, `postNavigationHooks`, `saveResponseCookies` (now `private readonly`; configure them through the constructor options as before), and the helpers `isRequestBlocked`, `applyCookies` (was `_applyCookies`), `handleNavigationTimeout` (was `_handleNavigationTimeout`), `throwIfProxyError` (was `_throwIfProxyError`)- **`BrowserLauncher`** — the helpers `getChromeExecutablePath`, `getTypicalChromeExecutablePath`, `validateProxyUrlProtocol` (were `_`-prefixed). `getDefaultHeadlessOption` (was `_getDefaultHeadlessOption`) stays `protected` — it is an override point (`PuppeteerLauncher` overrides it) — but lost its underscore prefix- **`RobotsTxtFile.load` and `Sitemap.parse`** — internal static factory helpers, now `private` (use the public `RobotsTxtFile.from` / `Sitemap.load` / `Sitemap.fromXmlString` entry points)-- Various internal fields on `BrowserController` (`id`, `browserPlugin`) and `BrowserPlugin` (`name`, `library`, `launchOptions`, `proxyUrl`, `userDataDir`, `browserPerProxy`, `ignoreProxyCertificate`) are now `readonly` (the `log` field on both is off the public surface entirely — see [`BrowserPool` internals are private](./browser_management.md#browserpool-internals-are-private))+- Various internal fields on `BrowserController` (`id`, `browserPlugin`) and `BrowserPlugin` (`name`, `library`, `launchOptions`, `proxyUrl`, `userDataDir`, `browserPerProxy`, `ignoreProxyCertificate`) are now `readonly` (the `log` field on both is off the public surface entirely — see [`BrowserPool` internals are private](./browser-management.md#browserpool-internals-are-private))The `running` and `hasFinishedBefore` flags on `BasicCrawler` were internal run-state bookkeeping for the re-run logic. If you were polling `crawler.running` to tell whether a crawl was in progress, track that yourself around the `crawler.run()` promise instead.diff --git a/docs/upgrading/upgrading_v4/removed_symbols.md b/docs/upgrading/upgrading-v4/removed-symbols.mdsimilarity index 93%rename from docs/upgrading/upgrading_v4/removed_symbols.mdrename to docs/upgrading/upgrading-v4/removed-symbols.mdindex db3ab8861..ce0f1496f 100644--- a/docs/upgrading/upgrading_v4/removed_symbols.md+++ b/docs/upgrading/upgrading-v4/removed-symbols.md@@ -6,7 +6,7 @@ sidebar_position: 10slug: /upgrading/upgrading-to-v4/removed-symbols----The full list of removed exports and members, for ctrl-F purposes. Where a replacement exists, it is noted inline. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+The full list of removed exports and members, for ctrl-F purposes. Where a replacement exists, it is noted inline. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Removed symbols@@ -30,16 +30,16 @@ The full list of removed exports and members, for ctrl-F purposes. Where a repla- `FileDownloadOptions.streamHandler` - streaming should now be handled directly in the `requestHandler` instead- `playwrightUtils.registerUtilsToContext` and `puppeteerUtils.registerUtilsToContext` - this is now added to the context via `ContextPipeline` composition- `context.blockResources` and `context.cacheResponses`, and the `puppeteerUtils.blockResources` / `puppeteerUtils.cacheResponses` functions behind them — both had a severe performance cost in recent Puppeteer versions and were already deprecated. Use `puppeteerUtils.blockRequests(page, options)`, which blocks URL patterns over CDP without disabling the browser cache. If you were caching responses, rely on the in-browser cache instead.-- `context.closeCookieModals`, `playwrightUtils.closeCookieModals` and `puppeteerUtils.closeCookieModals` — removed along with the optional `idcac-playwright` peer dependency (see [Crawling context no longer includes `closeCookieModals`](./upgrading_v4.md#crawling-context-no-longer-includes-closecookiemodals) and the [cookie modals guide](../../guides/cookie_modals.mdx))-- `Configuration.systemInfoV2` / `CRAWLEE_SYSTEM_INFO_V2` environment variable — the v2 behavior is now the default (see [Available resource detection](./utils_and_types.md#available-resource-detection))-- `KeyValueStore.getInput()` and `Configuration.inputKey` / `CRAWLEE_INPUT_KEY` — reading the run input moved to the Apify SDK (see [`KeyValueStore.getInput()` and `Configuration.inputKey` moved to the Apify SDK](./upgrading_v4.md#keyvaluestoregetinput-and-configurationinputkey-moved-to-the-apify-sdk))+- `context.closeCookieModals`, `playwrightUtils.closeCookieModals` and `puppeteerUtils.closeCookieModals` — removed along with the optional `idcac-playwright` peer dependency (see [Crawling context no longer includes `closeCookieModals`](./upgrading-v4.md#crawling-context-no-longer-includes-closecookiemodals) and the [cookie modals guide](../../guides/cookie-modals.mdx))+- `Configuration.systemInfoV2` / `CRAWLEE_SYSTEM_INFO_V2` environment variable — the v2 behavior is now the default (see [Available resource detection](./utils-and-types.md#available-resource-detection))+- `KeyValueStore.getInput()` and `Configuration.inputKey` / `CRAWLEE_INPUT_KEY` — reading the run input moved to the Apify SDK (see [`KeyValueStore.getInput()` and `Configuration.inputKey` moved to the Apify SDK](./upgrading-v4.md#keyvaluestoregetinput-and-configurationinputkey-moved-to-the-apify-sdk))- `Configuration.defaultDatasetId` / `defaultKeyValueStoreId` / `defaultRequestQueueId` and their `CRAWLEE_DEFAULT_*_ID` environment variables — the default storage is addressed by a reserved alias, not by a configurable ID. Open a storage by name if you need a specific one.-- `checkAndSerialize` and `chunkBySize` functions (from `@crawlee/core`) — value (de)serialization now lives in the `KeyValueStore` frontend; use `serializeValue` / `parseValue` (see [`maybeStringify` is removed](./storage_backends.md#maybestringify-is-removed))+- `checkAndSerialize` and `chunkBySize` functions (from `@crawlee/core`) — value (de)serialization now lives in the `KeyValueStore` frontend; use `serializeValue` / `parseValue` (see [`maybeStringify` is removed](./storage-backends.md#maybestringify-is-removed))- `BASIC_CRAWLER_TIMEOUT_BUFFER_SECS` constant (from `@crawlee/basic`) — was an internal timeout buffer, no longer exported-- `HttpResponse`, `HttpResponseWithoutBody`, `StreamingHttpResponse`, `ResponseTypes`, `BaseHttpResponseData`, `SimpleHeaders`, `processHttpRequestOptions`, and `GotScrapingHttpClient` (from `@crawlee/core`) — the HTTP client surface moved to `@crawlee/http-client` / `@crawlee/got-scraping-client` (see [HTTP client packages and `BaseHttpClient` reshaped](./http_clients.md#http-client-packages-and-basehttpclient-reshaped))-- `StreamHandlerContext` and `FileDownloadOptions` types (from `@crawlee/http`) — see [`FileDownload` now extends `BasicCrawler`](./upgrading_v4.md#filedownload-now-extends-basiccrawler-and-no-longer-takes-filedownloadoptions)-- `PlainResponse` type (from `@crawlee/http`) — it wrapped the `got-scraping` response and is gone along with the rest of the old HTTP response surface (see [`CrawlingContext.response` is now of type `Response`](./upgrading_v4.md#crawlingcontextresponse-is-now-of-type-response))-- `checkStorageAccess`, `withCheckedStorageAccess` and the `RequestHandlerResult` type — superseded by the storage transaction mechanism; use `withDirectStorageAccess()` and `StorageTransactionView` (see [Storage writes in request handlers are transactional](./upgrading_v4.md#storage-writes-in-request-handlers-are-transactional))+- `HttpResponse`, `HttpResponseWithoutBody`, `StreamingHttpResponse`, `ResponseTypes`, `BaseHttpResponseData`, `SimpleHeaders`, `processHttpRequestOptions`, and `GotScrapingHttpClient` (from `@crawlee/core`) — the HTTP client surface moved to `@crawlee/http-client` / `@crawlee/got-scraping-client` (see [HTTP client packages and `BaseHttpClient` reshaped](./http-clients.md#http-client-packages-and-basehttpclient-reshaped))+- `StreamHandlerContext` and `FileDownloadOptions` types (from `@crawlee/http`) — see [`FileDownload` now extends `BasicCrawler`](./upgrading-v4.md#filedownload-now-extends-basiccrawler-and-no-longer-takes-filedownloadoptions)+- `PlainResponse` type (from `@crawlee/http`) — it wrapped the `got-scraping` response and is gone along with the rest of the old HTTP response surface (see [`CrawlingContext.response` is now of type `Response`](./upgrading-v4.md#crawlingcontextresponse-is-now-of-type-response))+- `checkStorageAccess`, `withCheckedStorageAccess` and the `RequestHandlerResult` type — superseded by the storage transaction mechanism; use `withDirectStorageAccess()` and `StorageTransactionView` (see [Storage writes in request handlers are transactional](./upgrading-v4.md#storage-writes-in-request-handlers-are-transactional))- `CreateContextOptions` type (from `@crawlee/basic`) — a leftover of the pre-`ContextPipeline` context-creation design, unused by the library itself; context construction is now driven by `ContextPipeline`- `ResponseLike` interface (from `@crawlee/core`) — a vestige of the pre-`fetch` HTTP implementation with no consumers; `getCookiesFromResponse()` has always taken a native `Response`- `UrlPatternObject` (from `@crawlee/core`) — the *compiled* form of a URL pattern, produced internally by the `enqueueLinks()` machinery. Keep using `UrlPatternInput` / `GlobInput` / `RegExpInput`, which are unchanged, and let the return type of the pattern helpers be inferreddiff --git a/docs/upgrading/upgrading_v4/request_loaders.md b/docs/upgrading/upgrading-v4/request-loaders.mdsimilarity index 98%rename from docs/upgrading/upgrading_v4/request_loaders.mdrename to docs/upgrading/upgrading-v4/request-loaders.mdindex 3834b7f29..47d386bbf 100644--- a/docs/upgrading/upgrading_v4/request_loaders.md+++ b/docs/upgrading/upgrading-v4/request-loaders.md@@ -6,11 +6,11 @@ sidebar_position: 6slug: /upgrading/upgrading-to-v4/request-loaders----Applies when you use `RequestList` or `SitemapRequestList`, pass `requestList` / `requestQueue` to a crawler, or implement the request loader and manager interfaces yourself. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you use `RequestList` or `SitemapRequestList`, pass `requestList` / `requestQueue` to a crawler, or implement the request loader and manager interfaces yourself. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Request loaders and managers-The request loader/manager interfaces have been reworked. See the new [Request loaders](../../guides/request_loaders.mdx) guide for the full picture.+The request loader/manager interfaces have been reworked. See the new [Request loaders](../../guides/request-loaders.mdx) guide for the full picture.### `RequestQueue.addRequestsBatched` no longer retries rejected requests@@ -103,7 +103,7 @@ One probe instead of two, which a task loop runs several times a second, plus twIf you implemented either interface, return `ready` without evaluating anything further — it is the most common answer and the only one a caller can act on immediately. Reading from two sources, the precedence is `ready` > `stalled` > `waiting` > `finished`, and a combined `waiting` carries the earlier `readyAt`.-**Storage backends keep the two booleans** — see [`StorageBackend` interface simplified](./storage_backends.md#storagebackend-interface-simplified).+**Storage backends keep the two booleans** — see [`StorageBackend` interface simplified](./storage-backends.md#storagebackend-interface-simplified).### Combining a list and a queue: `toTandem()`@@ -173,7 +173,7 @@ const crawler = new CheerioCrawler({For the domains you list, a 429 is treated as a rate limit before `blockedStatusCodes` is consulted at all — it honours `Retry-After` (or backs off exponentially), holds only that domain's requests back, and leaves both the session and the request's retry budget untouched. Removing 429 from `blockedStatusCodes` therefore only affects domains the manager does not cover; you do not need to touch it to adopt throttling. Because those retries are free, a domain that never stops rate-limiting would keep the crawl alive indefinitely — so one that goes `maxDomainStallSecs` (15 minutes by default) without letting a single request through shuts the crawl down with a `PersistentRateLimitError`, leaving its requests queued for a later run — unless `keepAlive` is set, where outliving such a domain is the point.-It is also what enforces robots.txt `Crawl-delay` directives — with `respectRobotsTxtFile` enabled and no throttling manager covering the domain, the directive is ignored and the crawler warns about it. See the [request loaders guide](../../guides/request_loaders.mdx#per-domain-throttling).+It is also what enforces robots.txt `Crawl-delay` directives — with `respectRobotsTxtFile` enabled and no throttling manager covering the domain, the directive is ignored and the crawler warns about it. See the [request loaders guide](../../guides/request-loaders.mdx#per-domain-throttling).### `BasicCrawler.requestList` and `BasicCrawler.requestQueue` fields removeddiff --git a/docs/upgrading/upgrading_v4/sessions_and_proxies.md b/docs/upgrading/upgrading-v4/sessions-and-proxies.mdsimilarity index 98%rename from docs/upgrading/upgrading_v4/sessions_and_proxies.mdrename to docs/upgrading/upgrading-v4/sessions-and-proxies.mdindex f2eed4c5f..1a2116933 100644--- a/docs/upgrading/upgrading_v4/sessions_and_proxies.md+++ b/docs/upgrading/upgrading-v4/sessions-and-proxies.md@@ -6,13 +6,13 @@ sidebar_position: 2slug: /upgrading/upgrading-to-v4/sessions-and-proxies----Applies when you construct `SessionPool` or `Session` instances directly, implement your own pool, or configure proxies beyond a static URL list. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you construct `SessionPool` or `Session` instances directly, implement your own pool, or configure proxies beyond a static URL list. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Custom `SessionPool` implementations via the `ISessionPool` interfaceCrawlers now accept any object implementing the new `ISessionPool` interface as their `sessionPool` option, not just instances of the built-in `SessionPool`. The contract is intentionally tiny — a single method, `getSession(sessionId?)`, that hands out a session for a request. Lifecycle (reset, teardown) is the responsibility of whoever owns the pool: a custom pool you construct yourself is never owned by the crawler, so the crawler never tears it down. This makes it straightforward to plug in a remote, shared, or otherwise customized session-management strategy without subclassing `SessionPool` or copying its internals.-`ISessionPool` and `ISession` live in `@crawlee/types`; they are not re-exported from `@crawlee/core` (see [`@crawlee/types` symbols are no longer re-exported](./utils_and_types.md#crawleetypes-symbols-are-no-longer-re-exported)).+`ISessionPool` and `ISession` live in `@crawlee/types`; they are not re-exported from `@crawlee/core` (see [`@crawlee/types` symbols are no longer re-exported](./utils-and-types.md#crawleetypes-symbols-are-no-longer-re-exported)).```typescriptimport { BasicCrawler, Session } from '@crawlee/basic';@@ -142,7 +142,7 @@ More complex routing (more tiers, weighted draws, sticky assignment, cooldowns)### `ProxyConfiguration.newUrl` / `newProxyInfo` signatures changed-Because proxy tiers are gone, the leading `sessionId` positional argument was dropped from `ProxyConfiguration.newUrl()` and `ProxyConfiguration.newProxyInfo()`. `newUrl()` now takes no arguments, and `newProxyInfo()` takes only an optional, previously created `ProxyInfo` (e.g. one restored with a persisted session) instead of `(sessionId?, options?)`. Custom `IProxyConfiguration` implementations should return that `ProxyInfo` (refreshed if it depends on the environment) rather than a new one, otherwise the crawler replaces the session's proxy. The `ProxyConfigurationFunction` callback (the `newUrlFunction` option) was likewise simplified — it no longer receives any arguments, neither a `sessionId` nor a `{ request }` object. The proxy is resolved once per session, so to route specific requests through specific proxies, pin them to named sessions (see [Pinning a request to a specific session](../../guides/session_management.mdx#pinning-a-request-to-a-specific-session)). The `TieredProxy` interface and the `TieredProxyOptions` type have been removed.+Because proxy tiers are gone, the leading `sessionId` positional argument was dropped from `ProxyConfiguration.newUrl()` and `ProxyConfiguration.newProxyInfo()`. `newUrl()` now takes no arguments, and `newProxyInfo()` takes only an optional, previously created `ProxyInfo` (e.g. one restored with a persisted session) instead of `(sessionId?, options?)`. Custom `IProxyConfiguration` implementations should return that `ProxyInfo` (refreshed if it depends on the environment) rather than a new one, otherwise the crawler replaces the session's proxy. The `ProxyConfigurationFunction` callback (the `newUrlFunction` option) was likewise simplified — it no longer receives any arguments, neither a `sessionId` nor a `{ request }` object. The proxy is resolved once per session, so to route specific requests through specific proxies, pin them to named sessions (see [Pinning a request to a specific session](../../guides/session-management.mdx#pinning-a-request-to-a-specific-session)). The `TieredProxy` interface and the `TieredProxyOptions` type have been removed.**Before:**```typescriptdiff --git a/docs/upgrading/upgrading_v4/statistics.md b/docs/upgrading/upgrading-v4/statistics.mdsimilarity index 98%rename from docs/upgrading/upgrading_v4/statistics.mdrename to docs/upgrading/upgrading-v4/statistics.mdindex ed0b74eaa..84868b7b2 100644--- a/docs/upgrading/upgrading_v4/statistics.md+++ b/docs/upgrading/upgrading-v4/statistics.md@@ -6,7 +6,7 @@ sidebar_position: 4slug: /upgrading/upgrading-to-v4/statistics----Applies when you passed `statisticsOptions` to a crawler, subclassed `Statistics`, or passed type arguments to `BrowserCrawler`/`BrowserCrawlerOptions`. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you passed `statisticsOptions` to a crawler, subclassed `Statistics`, or passed type arguments to `BrowserCrawler`/`BrowserCrawlerOptions`. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## `statisticsOptions` is replaced by a `statistics` instance@@ -43,7 +43,7 @@ A persisted record is also validated on load now. One that does not match the ex## Subclassing `Statistics` to track extra fields is replaced by the `stateExtension` option-`persistStateKey` was `protected` and is now private. Declare extra fields via the new `stateExtension` option instead — `{ defaultState, deserialize, serialize }`, the same trio `RecoverableState` takes, scoped to the custom fields. See the [Custom statistics fields](../../guides/custom_statistics.mdx) guide.+`persistStateKey` was `protected` and is now private. Declare extra fields via the new `stateExtension` option instead — `{ defaultState, deserialize, serialize }`, the same trio `RecoverableState` takes, scoped to the custom fields. See the [Custom statistics fields](../../guides/custom-statistics.mdx) guide.The persisted record is now validated strictly, and keys that are neither built-in nor declared in `stateExtension` are dropped rather than written back. `calculate()` is still public and still an override point.diff --git a/docs/upgrading/upgrading_v4/storage_backends.md b/docs/upgrading/upgrading-v4/storage-backends.mdsimilarity index 99%rename from docs/upgrading/upgrading_v4/storage_backends.mdrename to docs/upgrading/upgrading-v4/storage-backends.mdindex a69f5aaf0..4a764b1af 100644--- a/docs/upgrading/upgrading_v4/storage_backends.md+++ b/docs/upgrading/upgrading-v4/storage-backends.md@@ -6,7 +6,7 @@ sidebar_position: 7slug: /upgrading/upgrading-to-v4/storage-backends----Applies when you implement your own storage backend, or reach below the `Dataset`, `KeyValueStore` and `RequestQueue` frontends. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you implement your own storage backend, or reach below the `Dataset`, `KeyValueStore` and `RequestQueue` frontends. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## `StorageBackend` interface simplified@@ -89,7 +89,7 @@ Methods that may have "nothing" to return now consistently resolve to `undefined- `isEmpty()` is the weak check — `true` when the next `fetchNextRequest()` would return `undefined`, i.e. there is nothing left to fetch right now. Requests that are currently in progress (fetched but not yet handled or reclaimed) are **not** counted, because they are not fetchable. This is what drives the crawler's task scheduling.- `isFinished()` is the strong check — `true` only when there are no pending requests **and** no requests currently in progress (including those locked by other clients sharing the queue). This is what determines whether crawling is actually done. An in-progress request keeps the queue *empty but not finished*, which is what stops a crawler from shutting down while a request is still being processed.-The loader and manager frontends do **not** draw that distinction — `IRequestLoader` and `IRequestManager` answer both questions with one [`checkReadiness()`](./request_loaders.md#isempty--isfinished-replaced-by-checkreadiness) call. The split lives at the backend boundary because that is the layer where the two questions really are two separate storage lookups; a frontend that split them too would either probe twice per scheduling decision or lose the distinction, whereas a backend that answers one at a time costs its caller nothing.+The loader and manager frontends do **not** draw that distinction — `IRequestLoader` and `IRequestManager` answer both questions with one [`checkReadiness()`](./request-loaders.md#isempty--isfinished-replaced-by-checkreadiness) call. The split lives at the backend boundary because that is the layer where the two questions really are two separate storage lookups; a frontend that split them too would either probe twice per scheduling decision or lose the distinction, whereas a backend that answers one at a time costs its caller nothing.The separate `RequestQueueV1`/`RequestQueueV2` classes (and the `RequestProvider` base class) have been removed. They no longer differ in behavior — request coordination is now internal to the storage backend — so they are merged into a single `RequestQueue` class. Replace any `RequestQueueV1`, `RequestQueueV2`, or `RequestProvider` imports with `RequestQueue`.diff --git a/docs/upgrading/upgrading_v4/upgrading_v4.md b/docs/upgrading/upgrading-v4/upgrading-v4.mdsimilarity index 98%rename from docs/upgrading/upgrading_v4/upgrading_v4.mdrename to docs/upgrading/upgrading-v4/upgrading-v4.mdindex 3c6cf3bd7..8679cf97c 100644--- a/docs/upgrading/upgrading_v4/upgrading_v4.md+++ b/docs/upgrading/upgrading-v4/upgrading-v4.md@@ -12,20 +12,20 @@ This page summarizes the breaking changes in Crawlee v4. There are many, so the- [Rename cheat sheet](#rename-cheat-sheet) — the purely mechanical renames, in one table.- [Changes most users will hit](#changes-most-users-will-hit) — read this part in full.- The **Only if you…** sections — each applies only if you use what its title says. Skim the titles and skip what doesn't concern you. The longer ones only summarize the changes here and link to a separate page with the details.-- [Appendix: removed symbols](./removed_symbols.md) — for when the compiler hands you a missing name and you want to know where it went.+- [Appendix: removed symbols](./removed-symbols.md) — for when the compiler hands you a missing name and you want to know where it went.## What v4 does better- **Timeouts that mean what they say.** Navigation and the request handler are [timed separately](#navigation-and-the-request-handler-are-timed-separately) — no more mysteriously summed limits — and a single route can get [its own timeout](#per-route-and-per-request-handler-timeouts) or extend it mid-flight.- **Composable crawling context.** The new `extendContext` option and `ContextPipeline` composition replace subclassing tricks for [adding members to the crawling context](#crawling-context-no-longer-includes-a-reference-to-the-crawler-itself).-- **Bring your own implementation.** Crawlers now accept any [`ISessionPool`](./sessions_and_proxies.md#custom-sessionpool-implementations-via-the-isessionpool-interface), [`IBrowserPool`](./browser_management.md#custom-browserpool-implementations-via-the-ibrowserpool-interface), [`IRenderingTypePredictor`](./browser_management.md#custom-rendering-type-predictors-via-the-irenderingtypepredictor-interface), [`IRequestManager`](./request_loaders.md#request-loaders-and-managers) or [`IStatistics`](./statistics.md#statisticsoptions-is-replaced-by-a-statistics-instance) — and never tear down an instance they did not create, which [`await using` now does for you](#collaborators-you-own-are-disposable).+- **Bring your own implementation.** Crawlers now accept any [`ISessionPool`](./sessions-and-proxies.md#custom-sessionpool-implementations-via-the-isessionpool-interface), [`IBrowserPool`](./browser-management.md#custom-browserpool-implementations-via-the-ibrowserpool-interface), [`IRenderingTypePredictor`](./browser-management.md#custom-rendering-type-predictors-via-the-irenderingtypepredictor-interface), [`IRequestManager`](./request-loaders.md#request-loaders-and-managers) or [`IStatistics`](./statistics.md#statisticsoptions-is-replaced-by-a-statistics-instance) — and never tear down an instance they did not create, which [`await using` now does for you](#collaborators-you-own-are-disposable).- **One concurrency budget for several crawlers.** The new [`ConcurrencySystem`](./autoscaling.md#autoscaling-moved-to-concurrencysystem) can be shared between crawlers, capping their combined concurrency instead of letting each one oversubscribe the host.-- **Native `fetch` types.** HTTP clients and `context.response` now use the [standard `Response`](#crawlingcontextresponse-is-now-of-type-response), and `got-scraping` is an [opt-in dependency](./http_clients.md#http-client-packages-and-basehttpclient-reshaped) instead of a mandatory one.-- **The session is the rotation unit.** A session carries its proxy, cookies and error score, and is rotated as a whole when blocked — replacing [proxy tiers](./sessions_and_proxies.md#tieredproxyurls-is-removed-from-proxyconfiguration) and [session rotation counters](#maxsessionrotations-and-requestsessionrotationcount-are-removed).+- **Native `fetch` types.** HTTP clients and `context.response` now use the [standard `Response`](#crawlingcontextresponse-is-now-of-type-response), and `got-scraping` is an [opt-in dependency](./http-clients.md#http-client-packages-and-basehttpclient-reshaped) instead of a mandatory one.+- **The session is the rotation unit.** A session carries its proxy, cookies and error score, and is rotated as a whole when blocked — replacing [proxy tiers](./sessions-and-proxies.md#tieredproxyurls-is-removed-from-proxyconfiguration) and [session rotation counters](#maxsessionrotations-and-requestsessionrotationcount-are-removed).- **Crawlers stop stepping on each other.** Multiple crawlers in one process [no longer share the default request queue](#multiple-crawler-instances-use-separate-default-request-queues), and repeated `run()` calls [no longer empty it](#repeated-run-calls-no-longer-empty-the-request-queue) behind your back.- **Cookies behave.** `sendRequest` finally [respects your `Cookie` header](#cookie-handling-in-httpcrawler-and-sendrequest), and browser cookies set inside the handler are [persisted to the session](#browser-cookies-are-also-persisted-after-requesthandler).- **No half-written results.** Storage writes in a request handler are [transactional](#storage-writes-in-request-handlers-are-transactional) — a handler that throws leaves nothing behind, and its retry does not duplicate data.-- **Simpler storage backend contract.** A custom storage backend is now [4 classes instead of 7](./storage_backends.md#storagebackend-interface-simplified).+- **Simpler storage backend contract.** A custom storage backend is now [4 classes instead of 7](./storage-backends.md#storagebackend-interface-simplified).## Rename cheat sheet@@ -40,17 +40,17 @@ The purely mechanical renames, collected in one place. Where a row links to a se| `config` (option / property name) | `configuration` ([details](#config-is-renamed-to-configuration-everywhere)) || `Configuration.getGlobalConfig()` | `Configuration.getGlobalConfiguration()` || `LocalEventManager.fromConfig()` | `LocalEventManager.fromConfiguration()` |-| `StorageClient` | `StorageBackend` ([reshaped](./storage_backends.md#storagebackend-interface-simplified)) |+| `StorageClient` | `StorageBackend` ([reshaped](./storage-backends.md#storagebackend-interface-simplified)) || `MemoryStorage` (`@crawlee/memory-storage`) | `MemoryStorageBackend` (`@crawlee/core`, [now memory-only](#memorystorage-split-into-filesystemstoragebackend-and-memorystoragebackend)) || `RequestQueueV1` / `RequestQueueV2` / `RequestProvider` | `RequestQueue` |-| `IRequestList` | `IRequestLoader` ([reshaped](./request_loaders.md#irequestlist-renamed-to-irequestloader)) |-| `SitemapRequestList` | `SitemapRequestLoader` ([details](./request_loaders.md#sitemaprequestlist-renamed-to-sitemaprequestloader)) |+| `IRequestList` | `IRequestLoader` ([reshaped](./request-loaders.md#irequestlist-renamed-to-irequestloader)) |+| `SitemapRequestList` | `SitemapRequestLoader` ([details](./request-loaders.md#sitemaprequestlist-renamed-to-sitemaprequestloader)) || `RobotsFile` | `RobotsTxtFile` || `markRequestHandled()` | `markRequestAsHandled()` || `requestList.length()` / `requestList.handledCount()` | `await getTotalCount()` / `await getHandledCount()` |-| `requestList.isEmpty()` / `requestList.isFinished()` | `await checkReadiness()` ([details](./request_loaders.md#isempty--isfinished-replaced-by-checkreadiness)) |-| `Dataset.listItems()` | `Dataset.getData()` / `Dataset.values()` ([details](./storage_backends.md#datasetlistitems-replaced-by-datasetgetdata-and-datasetvalues)) |-| crawler options `requestList` / `requestQueue` | `requestManager` ([details](./request_loaders.md#crawler-requestlist--requestqueue-options-deprecated-in-favor-of-requestmanager)) |+| `requestList.isEmpty()` / `requestList.isFinished()` | `await checkReadiness()` ([details](./request-loaders.md#isempty--isfinished-replaced-by-checkreadiness)) |+| `Dataset.listItems()` | `Dataset.getData()` / `Dataset.values()` ([details](./storage-backends.md#datasetlistitems-replaced-by-datasetgetdata-and-datasetvalues)) |+| crawler options `requestList` / `requestQueue` | `requestManager` ([details](./request-loaders.md#crawler-requestlist--requestqueue-options-deprecated-in-favor-of-requestmanager)) || `enqueueLinks({ requestQueue })` | `enqueueLinks({ requestManager })` || `enqueueLinks({ globs, regexps, pseudoUrls })` | `enqueueLinks({ include })` ([details](#globs-regexps-and-pseudourls-replaced-by-include)) || `(await enqueueLinks()).processedRequests` | `(await enqueueLinks()).addedRequests` ([details](#enqueuelinks-return-value-reshaped-addrequestsbatchedresult-instead-of-batchaddrequestsresult)) |@@ -250,7 +250,7 @@ const crawler = new PlaywrightCrawler({});```-`PuppeteerLauncher` keeps its `(launchContext?, configuration?)` constructor signature — this change is only about the crawler classes. `PlaywrightLauncher` is no longer exported at all, see [`PlaywrightLauncher` is no longer exported](./browser_management.md#playwrightlauncher-is-no-longer-exported).+`PuppeteerLauncher` keeps its `(launchContext?, configuration?)` constructor signature — this change is only about the crawler classes. `PlaywrightLauncher` is no longer exported at all, see [`PlaywrightLauncher` is no longer exported](./browser-management.md#playwrightlauncher-is-no-longer-exported).### Configuration class redesign@@ -1102,7 +1102,7 @@ With `respectRobotsTxtFile` enabled, a site that answers `/robots.txt` with a `5### `sameDomainDelaySecs` is now backed by `ThrottlingRequestManager`-`sameDomainDelaySecs` still works and still means what it did in v3 — subdomains included, it paces a whole registrable domain rather than a single host. Underneath, it is now a floor reported to the crawler's request manager as a [pacing signal](./request_loaders.md#irequestmanager-gained-recordpacingsignal); only when nothing there paces does the crawler wrap its manager in a `ThrottlingRequestManager`, which gives each domain a queue of its own so a delayed request waits in storage rather than in an in-memory map. Consequences worth knowing about:+`sameDomainDelaySecs` still works and still means what it did in v3 — subdomains included, it paces a whole registrable domain rather than a single host. Underneath, it is now a floor reported to the crawler's request manager as a [pacing signal](./request-loaders.md#irequestmanager-gained-recordpacingsignal); only when nothing there paces does the crawler wrap its manager in a `ThrottlingRequestManager`, which gives each domain a queue of its own so a delayed request waits in storage rather than in an in-memory map. Consequences worth knowing about:- A crawl that discovers more than `maxThrottledDomains` domains (100 by default) throws instead of quietly running out of steam. Pass your own `ThrottlingRequestManager` as `requestManager` to raise the ceiling — or crawl fewer sites.- Combined with a manager that paces on its own, a `ThrottlingRequestManager` with `domains: 'all'` and `throttleBy: 'registrableDomain'` takes the delay as its `minCrawlDelaySecs` floor, wherever it sits in a composition. One that paces only *some* domains throws instead — set `domains: 'all'`, or configure the delay there yourself and drop the option.@@ -1122,13 +1122,13 @@ See [Upgrading to v4: subclassing and internals](./internals.md) for the detailsApplies when you construct `SessionPool` or `Session` instances directly, implement your own pool, or configure proxies beyond a static URL list. Crawlers accept any `ISessionPool` implementation, `createSessionFunction` no longer receives the pool, and `Session` no longer references its pool or emits `sessionRetired`. `tieredProxyUrls` is gone in favor of named sessions, and `ProxyConfiguration.newUrl()` / `newProxyInfo()` dropped their `sessionId` argument.-See [Upgrading to v4: sessions and proxies](./sessions_and_proxies.md) for the details.+See [Upgrading to v4: sessions and proxies](./sessions-and-proxies.md) for the details.## Only if you customize browser managementApplies when you construct a `BrowserPool` yourself, reach for `browserController`, or tune `AdaptivePlaywrightCrawler` internals. Crawlers accept any `IBrowserPool`, `@crawlee/browser-pool` is no longer re-exported from `crawlee`, and `browserController` is gone from the crawling context. Puppeteer cookies are synced at the browser-context level, a rendering type predictor you pass in is yours to initialize, and several unused options, types and `BrowserPool` internals were removed.-See [Upgrading to v4: browser management](./browser_management.md) for the details.+See [Upgrading to v4: browser management](./browser-management.md) for the details.## Only if you customize crawler statistics@@ -1140,7 +1140,7 @@ See [Upgrading to v4: crawler statistics](./statistics.md) for the details.Applies when you implemented `BaseHttpClient` yourself, or imported `gotScraping` from `@crawlee/utils`. `BaseHttpClient` moved to `@crawlee/http-client` and is built around a single `fetch()` method returning a native `Response`. The `got-scraping` client lives in the opt-in `@crawlee/got-scraping-client` package, and `gotScraping` is no longer exported from `@crawlee/utils`.-See [Upgrading to v4: HTTP clients](./http_clients.md) for the details.+See [Upgrading to v4: HTTP clients](./http-clients.md) for the details.## Only if you use `FileDownload`@@ -1158,13 +1158,13 @@ The crawling context in the `FileDownload` crawler no longer includes the `body`Applies when you use `RequestList` or `SitemapRequestList`, pass `requestList` / `requestQueue` to a crawler, or implement the request loader and manager interfaces yourself. `IRequestList` is now `IRequestLoader` with async counters, `isEmpty()` / `isFinished()` collapsed into `checkReadiness()`, and `SitemapRequestList` is `SitemapRequestLoader`. Crawlers read from a single `requestManager` (the `requestList` / `requestQueue` options still work, but are deprecated), and the new opt-in `ThrottlingRequestManager` can back off per domain on HTTP 429.-See [Upgrading to v4: request loaders and managers](./request_loaders.md) for the details.+See [Upgrading to v4: request loaders and managers](./request-loaders.md) for the details.## Only if you implement a storage backendApplies when you implement your own storage backend, or reach below the `Dataset`, `KeyValueStore` and `RequestQueue` frontends. The `StorageBackend` interface (formerly `StorageClient`) needs four classes instead of seven, with async factory methods and slimmer sub-backends. `RequestQueueV1` / `RequestQueueV2` are merged into `RequestQueue`, the `requestLocking` experiment and `requestLockSecs` are gone, and `Dataset.listItems()` is replaced by `getData()` / `values()`.-See [Upgrading to v4: storage backends](./storage_backends.md) for the details.+See [Upgrading to v4: storage backends](./storage-backends.md) for the details.## Only if you tuned autoscaling@@ -1176,7 +1176,7 @@ See [Upgrading to v4: autoscaling](./autoscaling.md) for the details.Applies when you import utility functions, enums or types directly from `@crawlee/utils` or `@crawlee/types`, rather than only using the crawlers. The low-level resource-detection helpers are removed, `@crawlee/types` utility types are no longer re-exported from other packages, and several `@crawlee/utils` exports moved to the `@crawlee/utils/internal` entry point or were dropped. `RobotsTxtFile.find()` takes `proxyUrl` in its options, and the HTML-parsing helpers are now asynchronous.-See [Upgrading to v4: `@crawlee/utils` and `@crawlee/types`](./utils_and_types.md) for the details.+See [Upgrading to v4: `@crawlee/utils` and `@crawlee/types`](./utils-and-types.md) for the details.## Only if you use `JSDOMCrawler` or `LinkeDOMCrawler`@@ -1223,4 +1223,4 @@ A few Stagehand-specific option types were tightened:## Appendix: removed symbols-The full list of removed exports and members, for ctrl-F purposes, is on the [removed symbols](./removed_symbols.md) page.+The full list of removed exports and members, for ctrl-F purposes, is on the [removed symbols](./removed-symbols.md) page.diff --git a/docs/upgrading/upgrading_v4/utils_and_types.md b/docs/upgrading/upgrading-v4/utils-and-types.mdsimilarity index 99%rename from docs/upgrading/upgrading_v4/utils_and_types.mdrename to docs/upgrading/upgrading-v4/utils-and-types.mdindex b1da869f2..14c9a703c 100644--- a/docs/upgrading/upgrading_v4/utils_and_types.md+++ b/docs/upgrading/upgrading-v4/utils-and-types.md@@ -6,7 +6,7 @@ sidebar_position: 9slug: /upgrading/upgrading-to-v4/utils-and-types----Applies when you import utility functions, enums or types directly from `@crawlee/utils` or `@crawlee/types`, rather than only using the crawlers. This page is part of the [Upgrading to v4](./upgrading_v4.md) guide.+Applies when you import utility functions, enums or types directly from `@crawlee/utils` or `@crawlee/types`, rather than only using the crawlers. This page is part of the [Upgrading to v4](./upgrading-v4.md) guide.## Available resource detectiondiff --git a/oxlint.config.ts b/oxlint.config.tsindex 2600e2f9d..9de772b6d 100644--- a/oxlint.config.ts+++ b/oxlint.config.ts@@ -110,6 +110,7 @@ export default defineConfig({'import/no-default-export': 'off','unicorn/no-empty-file': 'off',+ 'unicorn/filename-case': ['error', { case: 'kebabCase' }],// Rules pulled in by `@apify/oxlint-config` that the codebase does not// currently satisfy. Disabled here to keep this migration scoped to a@@ -144,13 +145,6 @@ export default defineConfig({'vitest/hoisted-apis-on-top': 'off',},},- {- // docs are not migrated to kebab-case yet (#3113)- files: ['packages/**/*', 'test/**/*', 'scripts/**/*'],- rules: {- 'unicorn/filename-case': ['error', { case: 'kebabCase' }],- },- },{files: ['packages/templates/**/*'],rules: {