Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: Add a guide on ContextPipeline (#4232)

Jan Buchar45077502499d
diff --git a/docs/guides/crawling-context/crawling-context.mdx b/docs/guides/crawling-context/crawling-context.mdx
new file mode 100644
index 000000000..299eb0c11
--- /dev/null
+++ b/docs/guides/crawling-context/crawling-context.mdx
@@ -0,0 +1,80 @@
+---
+id: crawling-context
+title: Customizing the crawling context
+description: Extend the crawling context ad hoc or build custom crawlers on top of ContextPipeline
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+import ExtendContextSource from '!!raw-loader!./extend-context.ts';
+import CustomCrawlerSource from '!!raw-loader!./custom-crawler.ts';
+import HookReturnSource from '!!raw-loader!./hook-return.ts';
+
+The object your request handler receives (`request`, `session`, `page`, `$` and the rest) is built step by step. Every crawler runs each request through a chain of middlewares, the <ApiLink to="basic-crawler/class/ContextPipeline">`ContextPipeline`</ApiLink>, and each step adds members. The [Request lifecycle](./request-lifecycle) guide lists the steps and what happens around them. This guide is about adding your own.
+
+The [`extendContext` option](#extending-the-context-with-extendcontext) adds members to one crawler instance. A [crawler subclass](#writing-a-custom-crawler) with its own pipeline adds them to every instance of that class. A [navigation hook](#returning-from-a-navigation-hook) cannot add members, but it can replace existing ones.
+
+## Extending the context with `extendContext`
+
+When you need a helper or some shared state in your handlers, pass <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#extendContext">`extendContext`</ApiLink> when creating the crawler. It receives the context built so far and returns an object whose members get added to it:
+
+<CodeBlock className="language-typescript">{ExtendContextSource}</CodeBlock>
+
+The returned members are typed. `note` above is known to the `requestHandler`, the navigation hooks and the error handlers, and a typo in any of them is a compile-time error. The file handle is released with <ApiLink to="basic-crawler/interface/CrawlingContext#registerDeferredCleanup">`registerDeferredCleanup()`</ApiLink>, which runs once the request is done, whether it succeeded or failed (see [Cleanup](./request-lifecycle#cleanup) in the lifecycle guide).
+
+`extendContext` runs before navigation, so the members it returns are visible to `preNavigationHooks`, `postNavigationHooks` and the `requestHandler` alike. For the same reason its own input has no `page`, `response`, `$` or `body` yet. Anything that needs `page` or `response` can go in a `postNavigationHook`, and if `$` or `body` is required, use the request handler.
+
+Storage writes made from `extendContext` go through the same [storage transaction](./result-storage#transactional-storage) as the rest of the request, so a failed request rolls them back.
+
+### Returning from a navigation hook
+
+A `preNavigationHook` or `postNavigationHook` can also return a `Partial` of the crawling context, and the crawler copies the returned members onto it, replacing the existing ones. This is how `handleCloudflareChallengeHook()` works. After solving the challenge, the reloaded page's response replaces `response`, so the crawler's own blocked-request detection and the request handler both see it instead of the original 403. Written out by hand:
+
+<CodeBlock className="language-typescript">{HookReturnSource}</CodeBlock>
+
+Only existing members can be replaced this way. The handler's type does not change, so use `extendContext` or a subclass to add new ones.
+
+## Writing a custom crawler
+
+When the behavior should be reusable, with its own options, its own context type and its own setup and teardown, put it in a crawler class:
+
+<CodeBlock className="language-typescript">{CustomCrawlerSource}</CodeBlock>
+
+`TimedCrawler` is a `BasicCrawler` subclass with a single middleware. The middleware runs after `extendContext`, so `elapsedMillis` counts from there, not from when the request was fetched.
+
+The subclass passes `BasicCrawler` a <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#contextPipelineBuilder">`contextPipelineBuilder`</ApiLink>, a function that builds the crawler-specific part of the pipeline. `BasicCrawler` runs it after `extendContext`. The option is required whenever the `Context` type parameter is narrower than the default `CrawlingContext`, since without a pipeline the members declared by the type would never be added.
+
+### The pipeline API
+
+A <ApiLink to="basic-crawler/class/ContextPipeline">`ContextPipeline<TBase, TFinal>`</ApiLink> is an immutable chain of middlewares. `ContextPipeline.create<TBase>()` returns an empty one. `compose(middleware)` returns a new pipeline whose output type is this pipeline's output type plus the members the middleware returns. `chain(other)` appends another pipeline whose base type matches this one's output.
+
+A middleware is a function `(context, onCleanup) => extension`. The pipeline copies the returned `extension` onto the context object it received, so later middlewares and the request handler see one object with all the members. Property descriptors are copied as they are, so a getter in the extension stays a getter on the context.
+
+`onCleanup` registers teardown for anything the middleware sets up. Cleanups run once the request is done (see [Cleanup](./request-lifecycle#cleanup) in the lifecycle guide). Their argument is `undefined` when the request succeeded. If it failed, they receive the error instead, wrapped in a <ApiLink to="basic-crawler/class/ContextPipelineInitializationError">`ContextPipelineInitializationError`</ApiLink> or <ApiLink to="basic-crawler/class/RequestHandlerError">`RequestHandlerError`</ApiLink> with the original in `cause`. A cleanup should not throw, since that skips the remaining cleanups and stops the crawler.
+
+### Extending an existing crawler
+
+To add typed members on top of an existing crawler, extend `HttpCrawler`, override `buildContextPipeline()` and compose onto the parent's pipeline. In simplified form, `DOMCrawler` does this:
+
+```typescript
+protected override buildContextPipeline() {
+ return super
+ .buildContextPipeline()
+ .compose(this.#parseContent.bind(this))
+ .compose(this.#addHelpers.bind(this));
+}
+```
+
+`PlaywrightCrawler` and `PuppeteerCrawler` have a `buildContextPipeline()` too, and a subclass can override it the same way. There the result runs right after the page opens and before navigation, because `BrowserCrawler` adds the navigation steps after it. A step added this way has `page` but no `response`. Use a `postNavigationHook` for anything that needs the loaded page.
+
+If you ever need to check in the source code what a crawler does with a request, its pipeline is where to look. The built-in crawlers add all of their steps there, and the methods behind those steps are private, so a subclass changes them only by composing onto the pipeline.
+
+### Best practices
+
+- **Hide the `contextPipelineBuilder` crawler option unless the class is meant to be subclassed.** `CheerioCrawler`, `FileDownload` and the sample above omit it from their options and supply the builder themselves. `HttpCrawler` still accepts it, and a builder passed there replaces its whole pipeline, the HTTP request included. To extend `HttpCrawler`, override `buildContextPipeline()` instead, as shown above.
+- **Pass the extension type parameters through.** The crawler does not need to be generic over its `Context`, but it should take `ContextExtension` and `ExtendedContext` and forward them to the base class, as `TimedCrawler` does. That is what lets users of your crawler pass `extendContext` and still get a typed handler.
+- **Keep the step methods private, not protected.** Overriding a step in a subclass and composing onto the pipeline are two ways to change the same thing. Allow only the pipeline, so there is one place to look. (Passing a method to `compose` detaches it from `this`, so pass `this.#step.bind(this)` or an arrow function.)
+- **Return new members, don't mutate.** Build the extension as an object and return it. The returned type is what makes the next middleware and the handler type-check.
+
+If you are migrating from the `CrawlerExtension` class and `crawler.use()` of earlier versions, both were removed in v4 in favor of the mechanisms above. See the [removed symbols](../upgrading/upgrading-to-v4/removed-symbols) list in the upgrading guide.
diff --git a/docs/guides/crawling-context/custom-crawler.ts b/docs/guides/crawling-context/custom-crawler.ts
new file mode 100644
index 000000000..416fd1c23
--- /dev/null
+++ b/docs/guides/crawling-context/custom-crawler.ts
@@ -0,0 +1,42 @@
+import type { BasicCrawlerOptions, CleanupRegistrar, CrawlingContext } from 'crawlee';
+import { BasicCrawler, ContextPipeline } from 'crawlee';
+
+interface TimedCrawlingContext extends CrawlingContext {
+ elapsedMillis: () => number;
+}
+
+class TimedCrawler<
+ ContextExtension = Record<string, never>,
+ ExtendedContext extends TimedCrawlingContext = TimedCrawlingContext & ContextExtension,
+> extends BasicCrawler<TimedCrawlingContext, ContextExtension, ExtendedContext> {
+ constructor(
+ options: Omit<
+ BasicCrawlerOptions<TimedCrawlingContext, ContextExtension, ExtendedContext>,
+ 'contextPipelineBuilder'
+ > = {},
+ ) {
+ super({
+ ...options,
+ contextPipelineBuilder: () =>
+ ContextPipeline.create<CrawlingContext>().compose(this.#startTimer.bind(this)),
+ });
+ }
+
+ #startTimer(context: CrawlingContext, onCleanup: CleanupRegistrar) {
+ const startedAt = Date.now();
+
+ onCleanup((error) => {
+ context.log.info(`${context.request.url} took ${Date.now() - startedAt} ms`, { failed: !!error });
+ });
+
+ return { elapsedMillis: () => Date.now() - startedAt };
+ }
+}
+
+const crawler = new TimedCrawler({
+ async requestHandler({ request, elapsedMillis, log }) {
+ log.info(`Hello from ${request.url} after ${elapsedMillis()}ms since the start of processing`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev', 'https://crawlee.dev/js']);
diff --git a/docs/guides/crawling-context/extend-context.ts b/docs/guides/crawling-context/extend-context.ts
new file mode 100644
index 000000000..0e8c44124
--- /dev/null
+++ b/docs/guides/crawling-context/extend-context.ts
@@ -0,0 +1,24 @@
+import { open } from 'node:fs/promises';
+
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Runs before navigation, so `$` and `response` are not available here yet.
+ extendContext: async (context) => {
+ const file = await open(`${context.request.url.replace(/[^a-z0-9]/gi, '_')}.log`, 'a');
+ // Closed once the request is done, including after a failure.
+ context.registerDeferredCleanup(() => file.close());
+
+ return {
+ async note(line: string) {
+ await file.appendFile(`${line}\n`);
+ },
+ };
+ },
+ async requestHandler({ note, $, enqueueLinks }) {
+ await note(`title: ${$('title').text()}`);
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/crawling-context/hook-return.ts b/docs/guides/crawling-context/hook-return.ts
new file mode 100644
index 000000000..3aa38ec39
--- /dev/null
+++ b/docs/guides/crawling-context/hook-return.ts
@@ -0,0 +1,21 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ postNavigationHooks: [
+ async ({ response, handleCloudflareChallenge, log }) => {
+ // 403 when the page is a Cloudflare challenge
+ log.info(`Navigation response: ${response?.status()}`);
+
+ const solved = await handleCloudflareChallenge();
+ // Replace `response` on the context, but only when a challenge was actually solved.
+ // `{ response: undefined }` would overwrite the navigation response.
+ return solved && { response: solved };
+ },
+ ],
+ async requestHandler({ response, log }) {
+ // 200 - the response the hook returned
+ log.info(`Request handler response: ${response?.status()}`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/request-lifecycle/request-lifecycle.mdx b/docs/guides/request-lifecycle/request-lifecycle.mdx
new file mode 100644
index 000000000..cce3f89f9
--- /dev/null
+++ b/docs/guides/request-lifecycle/request-lifecycle.mdx
@@ -0,0 +1,92 @@
+---
+id: request-lifecycle
+title: Request lifecycle
+description: What happens to a request between the queue and the request handler, and what happens when it fails
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+Every request a crawler picks up goes through the same sequence of steps before the `requestHandler` sees it, and through another fixed sequence afterwards, whether the handler succeeded or not. This guide covers where the navigation hooks run, what the timeouts cover, how retries and the error handlers work, how to skip a request on purpose, and when cleanup happens.
+
+## The steps
+
+Every crawler runs each request through a chain of middlewares, the <ApiLink to="basic-crawler/class/ContextPipeline">`ContextPipeline`</ApiLink>. Each step adds members to the crawling context the request handler eventually receives. `BasicCrawler` owns the first steps and the last one, the crawler class (`CheerioCrawler`, `PlaywrightCrawler`, …) contributes the middle, and the `extendContext` option sits between them. Dashed nodes are your code:
+
+```mermaid
+flowchart TB
+ subgraph basic ["BasicCrawler"]
+ direction LR
+ robots["robots.txt check"] --> base["base context<br/>(log, pushData, useState, …)"] --> session["session + proxyInfo"] --> helpers["addRequests, sendRequest"]
+ end
+ subgraph crawler ["Crawler class"]
+ direction LR
+ prepare["prepare navigation"] --> pre["preNavigationHooks"] --> nav["navigation"] --> post["postNavigationHooks"] --> parse["response processing, parsing, …"]
+ end
+ subgraph basic2 ["BasicCrawler"]
+ strategy["enqueue-strategy check"]
+ end
+ basic --> extend["extendContext"] --> crawler --> basic2 --> handler["requestHandler"]
+ classDef user stroke-dasharray: 5 5
+ class extend,pre,post,handler user
+```
+
+For `CheerioCrawler` the middle part makes the HTTP request and parses the body into `$`. For `PlaywrightCrawler` it opens a page and navigates it. The enqueue-strategy check skips requests that no longer match `strategy` after a redirect.
+
+To add your own members to the context, see [Customizing the crawling context](./crawling-context).
+
+## Navigation hooks
+
+`preNavigationHooks` run after `extendContext` and before the navigation, `postNavigationHooks` right after it. They receive the same context the request handler will, minus whatever navigation has not produced yet. A pre-navigation hook on a browser crawler has `page` but no `response`. On an HTTP crawler it has neither `response` nor `body`.
+
+A hook can return a `Partial` of the crawling context, and the crawler copies the returned members onto it, replacing the existing ones. This is how `handleCloudflareChallengeHook()` swaps the challenge response for the real one. See [Customizing the crawling context](./crawling-context#returning-from-a-navigation-hook) for an example.
+
+## Timeouts
+
+Three timeouts apply:
+
+| Option | Covers | Default |
+| --- | --- | --- |
+| `navigationTimeoutSecs` | the `preNavigationHooks`, the navigation, and the `postNavigationHooks` together | 30 (HTTP), 60 (browser) |
+| `requestHandlerTimeoutSecs` | the `requestHandler` only | 60 |
+| `CRAWLEE_INTERNAL_TIMEOUT` (env, ms) | the whole request, including the phases that have no timeout of their own (`extendContext`, the robots.txt check, response processing) | twice the request handler timeout, at least 5 minutes |
+
+`navigationTimeoutSecs` is a single budget shared by the whole navigation phase, so a slow hook eats into the window the navigation itself uses. A single route can override `requestHandlerTimeoutSecs` through its `router.addHandler()` options, and `context.extendTimeout(secs)` extends the current window from inside a hook or the handler.
+
+## Failures and retries
+
+A request handler, navigation hook, or any other step that throws fails the current attempt. The remaining steps and the request handler do not run. What happens next depends on whether the request can be retried.
+
+- The crawler retries while `request.retryCount` is below `maxRequestRetries` (or the request's own `maxRetries`), unless `request.noRetry` is set or the error is a <ApiLink to="core/class/NonRetryableError">`NonRetryableError`</ApiLink>. Throwing a <ApiLink to="basic-crawler/class/RetryRequestError">`RetryRequestError`</ApiLink> forces a retry regardless of the count.
+- Before each retry, the crawler calls `errorHandler` with the context and the error, then puts the request back in the queue.
+- When no retry is left, it calls `failedRequestHandler` instead, marks the request as handled and counts it as failed.
+- A <ApiLink to="core/class/CriticalError">`CriticalError`</ApiLink> stops the whole crawler.
+
+Both handlers may run after a failure that happened before the context was fully built, so everything the crawler class and `extendContext` add is typed as optional there. Check for `page` or `$` before using it. `request`, `log` and the storage helpers are always present. `session` is typed as present but is missing when fetching the session itself failed.
+
+A failed attempt also marks the session as bad, and a <ApiLink to="core/class/SessionError">`SessionError`</ApiLink> retires it. See [Session management](./session-management).
+
+### Skipping a request without failing it
+
+Some requests should end without counting as a failure. Calling <ApiLink to="basic-crawler/interface/RestrictedCrawlingContext#skipRequest">`context.skipRequest(message?)`</ApiLink> ends the request as skipped. The crawler marks it as handled, does not retry it, does not call `failedRequestHandler` and logs no error. <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#onSkippedRequest">`onSkippedRequest`</ApiLink> fires with the `'manual'` reason and the message you passed. It works from `extendContext`, a navigation hook, the request handler and `errorHandler`. In `failedRequestHandler` it has no effect.
+
+Storage writes made for the request before the skip are rolled back, the same as on failure.
+
+See the [Skipping requests](../examples/skip-request) example.
+
+## Storage writes
+
+Every request runs inside a storage transaction. Writes made through the context helpers or the storage classes are recorded while the request is being handled, applied all together when the request handler succeeds, and dropped when it fails or is skipped. A retry therefore never double-writes. See [Transactional storage](./result-storage#transactional-storage) for the details and the exceptions.
+
+## Cleanup
+
+Once the request handler has returned or thrown and any error handler has run, the crawler tears down what the steps set up, such as a browser page or an HTTP connection, and anything you registered yourself.
+
+You register teardown from `extendContext`, a navigation hook or the request handler with <ApiLink to="basic-crawler/interface/CrawlingContext#registerDeferredCleanup">`context.registerDeferredCleanup()`</ApiLink>. The callback runs once the request is done, whether it succeeded or failed. It receives no argument, and if it throws, the crawler logs the error at debug level and carries on.
+
+When they run:
+
+- Cleanups run after `errorHandler` and `failedRequestHandler`, so an error handler still sees the page or connection.
+- They run after the request's storage writes have been committed or rolled back. A storage write made in a cleanup applies immediately and is never rolled back.
+- Cleanups run in reverse registration order.
+
+Crawler subclasses register cleanup from their pipeline steps with the `onCleanup` callback instead, which also receives the failure. See [The pipeline API](./crawling-context#the-pipeline-api).
diff --git a/website/sidebars.js b/website/sidebars.js
index a1341d407..56f5912b2 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -55,6 +55,8 @@ module.exports = {
'guides/running-in-web-server/running-in-web-server',
'guides/parallel-scraping/parallel-scraping-guide',
'guides/custom-statistics',
+ 'guides/request-lifecycle/request-lifecycle',
+ 'guides/crawling-context/crawling-context',
'guides/custom-http-client/custom-http-client',
'guides/custom-logger/custom-logger'
],