Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: fix missing sidebar in web server guide (#2927)
Martin Adámekbbb13d47bc97
diff --git a/docs/guides/running-in-web-server/running-in-web-server.mdx b/docs/guides/running-in-web-server/running-in-web-server.mdxindex cee0f83a9..f12038e0f 100644--- a/docs/guides/running-in-web-server/running-in-web-server.mdx+++ b/docs/guides/running-in-web-server/running-in-web-server.mdx@@ -1,13 +1,11 @@----id: running-in-web-server-guide-title: Running in web server Guide-sidebar_label: Running in web server+id: running-in-web-server+title: Running in web serverdescription: Run Crawlee in web server using a request/response approach---import CodeBlock from '@theme/CodeBlock';import ApiLink from '@site/src/components/ApiLink';-import WebServerSource from '!!raw-loader!./web-server.mjs';Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.diff --git a/website/sidebars.js b/website/sidebars.jsindex 0ac1feb59..1ece79016 100644--- a/website/sidebars.js+++ b/website/sidebars.js@@ -44,6 +44,7 @@ module.exports = {'guides/got-scraping','guides/typescript-project','guides/docker-images',+ 'guides/running-in-web-server/running-in-web-server','guides/parallel-scraping/parallel-scraping-guide','guides/custom-http-client/custom-http-client'],diff --git a/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdxindex cee0f83a9..f12038e0f 100644--- a/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.11/guides/running-in-web-server/running-in-web-server.mdx@@ -1,13 +1,11 @@----id: running-in-web-server-guide-title: Running in web server Guide-sidebar_label: Running in web server+id: running-in-web-server+title: Running in web serverdescription: Run Crawlee in web server using a request/response approach---import CodeBlock from '@theme/CodeBlock';import ApiLink from '@site/src/components/ApiLink';-import WebServerSource from '!!raw-loader!./web-server.mjs';Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.diff --git a/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdxindex cee0f83a9..f12038e0f 100644--- a/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.12/guides/running-in-web-server/running-in-web-server.mdx@@ -1,13 +1,11 @@----id: running-in-web-server-guide-title: Running in web server Guide-sidebar_label: Running in web server+id: running-in-web-server+title: Running in web serverdescription: Run Crawlee in web server using a request/response approach---import CodeBlock from '@theme/CodeBlock';import ApiLink from '@site/src/components/ApiLink';-import WebServerSource from '!!raw-loader!./web-server.mjs';Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.diff --git a/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx b/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdxindex cee0f83a9..f12038e0f 100644--- a/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx+++ b/website/versioned_docs/version-3.13/guides/running-in-web-server/running-in-web-server.mdx@@ -1,13 +1,11 @@----id: running-in-web-server-guide-title: Running in web server Guide-sidebar_label: Running in web server+id: running-in-web-server+title: Running in web serverdescription: Run Crawlee in web server using a request/response approach---import CodeBlock from '@theme/CodeBlock';import ApiLink from '@site/src/components/ApiLink';-import WebServerSource from '!!raw-loader!./web-server.mjs';Most of the time, Crawlee jobs are run as batch jobs. You have a list of URLs you want to scrape every week or you might want to scrape a whole website once per day. After the scrape, you send the data to your warehouse for analytics. Batch jobs are efficient because they can use [Crawlee's built-in autoscaling](https://crawlee.dev/js/docs/guides/scaling-crawlers) to fully utilize the resources you have available. But sometimes you have a use-case where you need to return scrape data as soon as possible. There might be a user waiting on the other end so every millisecond counts. This is where running Crawlee in a web server comes in.diff --git a/website/versioned_sidebars/version-3.11-sidebars.json b/website/versioned_sidebars/version-3.11-sidebars.jsonindex 5ca179028..25d2250d7 100644--- a/website/versioned_sidebars/version-3.11-sidebars.json+++ b/website/versioned_sidebars/version-3.11-sidebars.json@@ -46,6 +46,7 @@"guides/got-scraping","guides/typescript-project","guides/docker-images",+ "guides/running-in-web-server/running-in-web-server","guides/parallel-scraping/parallel-scraping-guide"]},diff --git a/website/versioned_sidebars/version-3.12-sidebars.json b/website/versioned_sidebars/version-3.12-sidebars.jsonindex 8022872dd..f7bd1d6ed 100644--- a/website/versioned_sidebars/version-3.12-sidebars.json+++ b/website/versioned_sidebars/version-3.12-sidebars.json@@ -46,6 +46,7 @@"guides/got-scraping","guides/typescript-project","guides/docker-images",+ "guides/running-in-web-server/running-in-web-server","guides/parallel-scraping/parallel-scraping-guide","guides/custom-http-client/custom-http-client"]diff --git a/website/versioned_sidebars/version-3.13-sidebars.json b/website/versioned_sidebars/version-3.13-sidebars.jsonindex 8022872dd..f7bd1d6ed 100644--- a/website/versioned_sidebars/version-3.13-sidebars.json+++ b/website/versioned_sidebars/version-3.13-sidebars.json@@ -46,6 +46,7 @@"guides/got-scraping","guides/typescript-project","guides/docker-images",+ "guides/running-in-web-server/running-in-web-server","guides/parallel-scraping/parallel-scraping-guide","guides/custom-http-client/custom-http-client"]