Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Fixed Puppeteer crawler example

Jan Curne53e5e2d6edb
diff --git a/examples/puppeteer_crawler.js b/examples/puppeteer_crawler.js
index ccd13afb9..8e3767f35 100644
--- a/examples/puppeteer_crawler.js
+++ b/examples/puppeteer_crawler.js
@@ -14,7 +14,7 @@ Apify.main(async () => {
// Create and initialize an instance of the RequestList class that contains the start URL.
const requestList = new Apify.RequestList({
sources: [
- { url: 'http://www.ycombinator.com/' },
+ { url: 'https://news.ycombinator.com/' },
],
});
await requestList.initialize();
@@ -32,7 +32,10 @@ Apify.main(async () => {
// Here you can set options that are passed to the Apify.launchPuppeteer() function.
// For example, you can set "slowMo" to slow down Puppeteer operations to simplify debugging
- launchPuppeteerOptions: { slowMo: 2000 },
+ launchPuppeteerOptions: { slowMo: 500 },
+
+ // Stop crawling after several pages
+ maxRequestsPerCrawl: 10,
// This function will be called for each URL to crawl.
// Here you can write the Puppeteer scripts you are familiar with,