Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved puppeteer_crawler.js example
Jan Curn6bf89fdca21f
diff --git a/examples/puppeteer_crawler.js b/examples/puppeteer_crawler.jsindex c39572402..6e0c83176 100644--- a/examples/puppeteer_crawler.js+++ b/examples/puppeteer_crawler.js@@ -69,17 +69,14 @@ Apify.main(async () => {// Store the results to the default dataset.await Apify.pushData(data);- // Find the link to the next page using Puppeteer functions.- let nextHref;- try {- nextHref = await page.$eval('.morelink', el => el.href);- } catch (err) {- console.log(`${request.url} is the last page!`);- return;- }-- // Enqueue the link to the RequestQueue- await requestQueue.addRequest(new Apify.Request({ url: nextHref }));+ // Find a link to the next page and enqueue it if it exists.+ const infos = await Apify.utils.puppeteer.enqueueLinks({+ page,+ requestQueue,+ selector: '.morelink',+ });++ if (infos.length === 0) console.log(`${request.url} is the last page!`);},// This function is called if the page processing failed more than maxRequestRetries+1 times.