Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Add nullification of loadedUrl to PupCrw

Ondra Urbandeceabe09cb2
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index c19567344..ce3d1d037 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -292,6 +292,10 @@ class PuppeteerCrawler {
async _handleRequestFunction({ request, autoscaledPool }) {
const page = await this.puppeteerPool.newPage();
try {
+ // Delete loadedUrl before navigation to prevent user
+ // from getting an old loadedUrl in his custom gotoFunction
+ request.loadedUrl = null;
+
const response = await this.gotoFunction({ page, request, autoscaledPool, puppeteerPool: this.puppeteerPool });
request.loadedUrl = page.url();
await addTimeoutToPromise(