Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: Fix timeouts and retry limit exceeded in interim introduction example (#2791)

Angie Byron1ea95f74ae1f
diff --git a/docs/introduction/05-crawling.mdx b/docs/introduction/05-crawling.mdx
index 6621ee10b..a3930c6b6 100644
--- a/docs/introduction/05-crawling.mdx
+++ b/docs/introduction/05-crawling.mdx
@@ -23,16 +23,21 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
- // Wait for the category cards to render,
- // otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.collection-block-item');
-
- // Add links to the queue, but only from
- // elements matching the provided selector.
- await enqueueLinks({
- selector: '.collection-block-item',
- label: 'CATEGORY',
- });
+
+ // Only run this logic on the main category listing, not on sub-pages.
+ if (request.label !== 'CATEGORY') {
+
+ // Wait for the category cards to render,
+ // otherwise enqueueLinks wouldn't enqueue anything.
+ await page.waitForSelector('.collection-block-item');
+
+ // Add links to the queue, but only from
+ // elements matching the provided selector.
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
},
});
diff --git a/website/versioned_docs/version-3.12/introduction/05-crawling.mdx b/website/versioned_docs/version-3.12/introduction/05-crawling.mdx
index 6621ee10b..a3930c6b6 100644
--- a/website/versioned_docs/version-3.12/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.12/introduction/05-crawling.mdx
@@ -23,16 +23,21 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
- // Wait for the category cards to render,
- // otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.collection-block-item');
-
- // Add links to the queue, but only from
- // elements matching the provided selector.
- await enqueueLinks({
- selector: '.collection-block-item',
- label: 'CATEGORY',
- });
+
+ // Only run this logic on the main category listing, not on sub-pages.
+ if (request.label !== 'CATEGORY') {
+
+ // Wait for the category cards to render,
+ // otherwise enqueueLinks wouldn't enqueue anything.
+ await page.waitForSelector('.collection-block-item');
+
+ // Add links to the queue, but only from
+ // elements matching the provided selector.
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
},
});