Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Add nullification of loadedUrl to PupCrw
Ondra Urbandeceabe09cb2
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex c19567344..ce3d1d037 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -292,6 +292,10 @@ class PuppeteerCrawler {async _handleRequestFunction({ request, autoscaledPool }) {const page = await this.puppeteerPool.newPage();try {+ // Delete loadedUrl before navigation to prevent user+ // from getting an old loadedUrl in his custom gotoFunction+ request.loadedUrl = null;+const response = await this.gotoFunction({ page, request, autoscaledPool, puppeteerPool: this.puppeteerPool });request.loadedUrl = page.url();await addTimeoutToPromise(