Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: Add miscellaneous examples (#4247)

Jan Buchar04ffaba30fb1
diff --git a/docs/examples/stop-crawler.mdx b/docs/examples/stop-crawler.mdx
new file mode 100644
index 000000000..473ab5b4c
--- /dev/null
+++ b/docs/examples/stop-crawler.mdx
@@ -0,0 +1,16 @@
+---
+id: stop-crawler
+title: Stopping the crawler early
+---
+
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
+import ApiLink from '@site/src/components/ApiLink';
+import StopCrawlerSource from '!!raw-loader!roa-loader!./stop-crawler.ts';
+
+Calling <ApiLink to="basic-crawler/class/BasicCrawler#stop">`crawler.stop()`</ApiLink> ends the run gracefully: no further requests are dispatched, the ones already being processed finish, and `crawler.run()` resolves. Requests still waiting in the queue are left there untouched, so a later `run()` picks up where this one left off. This is the way to stop once a result limit is reached, without draining the queue or marking the remaining requests as handled.
+
+To abort immediately without waiting for in-flight requests, use <ApiLink to="basic-crawler/class/BasicCrawler#teardown">`crawler.teardown()`</ApiLink> instead.
+
+<RunnableCodeBlock className="language-js" type="cheerio">
+ {StopCrawlerSource}
+</RunnableCodeBlock>
diff --git a/docs/examples/stop-crawler.ts b/docs/examples/stop-crawler.ts
new file mode 100644
index 000000000..7f20d3acc
--- /dev/null
+++ b/docs/examples/stop-crawler.ts
@@ -0,0 +1,30 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks, pushData, useState }) {
+ const state = await useState({ found: 0, limit: 10 });
+
+ const price = $('.price').first().text();
+ if (price) {
+ await pushData({ url: request.url, price });
+ state.found++;
+ }
+
+ // Once we have what we came for, stop dispatching new requests. Requests already in
+ // flight finish normally and `crawler.run()` resolves; whatever is still queued stays queued.
+ if (state.found >= state.limit) {
+ crawler.stop('Collected enough prices');
+ return;
+ }
+
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// Calling `run()` again with no arguments continues with the requests left in the queue.
+// Here we raise the limit first, otherwise the first request would stop the crawler again.
+const state = await crawler.useState({ found: 0, limit: 10 });
+state.limit = 20;
+await crawler.run();
diff --git a/docs/guides/proxy-management.mdx b/docs/guides/proxy-management.mdx
index 873dba333..bbdfe6130 100644
--- a/docs/guides/proxy-management.mdx
+++ b/docs/guides/proxy-management.mdx
@@ -118,6 +118,25 @@ The `newUrlFunction` takes no arguments and returns a string with the proxy URL
Our crawlers will now use the selected proxies for all connections.
+### Proxies in browser crawlers
+
+A browser is launched with a single proxy, so in `PlaywrightCrawler` and `PuppeteerCrawler` the session's proxy only applies to a page if the browser it lands in was launched with the same one. By default, the pool opens the page in any browser with free capacity, which can ignore the session's proxy. Two `launchContext` options fix that:
+
+- `useIncognitoPages: true` - every page gets its own browser context with the session's proxy applied to it. Pages share no cookies or cache with each other; the session's cookies are still carried over.
+- `browserPerProxy: true` - the pool only reuses a browser that was launched with the page's proxy, and launches a new one otherwise. Useful when you need a shared context per proxy, at the cost of more browser instances.
+
+```js
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ proxyConfiguration,
+ launchContext: {
+ browserPerProxy: true,
+ },
+ // ...
+});
+```
+
## IP Rotation and session management
Each call to <ApiLink to="core/class/ProxyConfiguration#newUrl">`proxyConfiguration.newUrl()`</ApiLink> generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and <ApiLink to="basic-crawler/class/SessionPool">`SessionPool`</ApiLink> class for more information on how keeping a real session helps us avoid blocking.
diff --git a/docs/guides/result-storage.mdx b/docs/guides/result-storage.mdx
index 9e21c17eb..280bdea9e 100644
--- a/docs/guides/result-storage.mdx
+++ b/docs/guides/result-storage.mdx
@@ -10,6 +10,19 @@ Crawlee has several result storage types that are useful for specific tasks. The
By default, Crawlee storage is managed by the <ApiLink to="fs-storage/class/FileSystemStorageBackend">`FileSystemStorageBackend`</ApiLink> class, which stores all information as local files in the respective storage type folders.
+To keep everything in memory and never touch the disk - handy in tests or short-lived scripts - pass a <ApiLink to="core/class/MemoryStorageBackend">`MemoryStorageBackend`</ApiLink> instead. Both backends are exported from `crawlee`:
+
+```js
+import { CheerioCrawler, MemoryStorageBackend } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ storageBackend: new MemoryStorageBackend(),
+ // ...
+});
+```
+
+Setting `CRAWLEE_PERSIST_STORAGE=false` has the same effect globally, without touching the code.
+
## Key-value store
The key-value store is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.