Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

1.1 KB
Raw Blame
---
id: skip-navigation
title: Skipping navigations for certain requests
---

import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import SkipNavigationSource from '!!raw-loader!roa-loader!./skip-navigation.ts';

While crawling a website, you may encounter certain resources you'd like to save, but don't need the full power of a crawler to do so (like images delivered through a CDN).

By combining the <ApiLink to="core/class/Request#skipNavigation">`Request#skipNavigation`</ApiLink> option with <ApiLink to="basic-crawler/interface/BasicCrawlingContext#sendRequest">`sendRequest`</ApiLink>, we can fetch the image from the CDN, and save it to our key-value store without needing to use the full crawler.


:::info

For this example, we are using the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to showcase this, but this is available on all the crawlers we provide.

:::

<RunnableCodeBlock className="language-js">
	{SkipNavigationSource}
</RunnableCodeBlock>